nee.lv

Your second paragraph does not follow, at all, from the first. These are completely orthogonal demands. The gaming industry is absolutely overwhelmed with outrageously inefficient, garbage, crash-prone code. It has become the norm, and it has absolutely nothing to do with AI. Like https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/. That something so outrageously trash made it to a hundreds-of-million dollar game, cursing millions to 10+ minute waits, should shame everyone involved. It's actually completely normal in that industry. Trash code, thoughtless and lazily implemented, is the norm. Most game studios would likely hugely improve their game, har har, if they leveraged AI a lot more.

If this article was exciting for you, I also highly recommend this one. A random dude fixed a bug in GTA 5 that was the root cause of it loading insanely slowly since the game came out! https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

This was a good read and great work. Can't wait to see the performance tests. Your write up connected some early knowledge from when I was 11 where I was trying to set up a database/backend and was finding lots of cgi-bin online. I realize now those were spinning up new processes with each request https://en.wikipedia.org/wiki/Common_Gateway_Interface I remember when sendfile became available for my large gaming forum with dozens of TB of demo downloads. That alone was huge for concurrency. I thought I had swore off this type of engineering but between this, the Netflix case of extra 40ms and the GTA 5 70% load time reduction maybe there is a lot more impactful work to be done. https://netflixtechblog.com/life-of-a-netflix-partner-engineer-the-cas… https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

My favorite example of this is the GTA Online insane loading time issue that ended up being due to poor handling of a 10MB json file (and was finally tracked down by someone outside their org). Took a 6 minute load time down to just under 2 minutes: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

This was a great read. I wonder if the reason for the game client surprise crashing could be investigated the same way? Do the executable integrity checks run during runtime? What if one of those fails while it’s running, do you get a different disconnection message? I feel like this should be rewarded in the same way as the GTA V takes forever to load-writeup. https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

They can get great people but then choose to spend their effort solely on features and ignoring things like size and speed - for example, the GTA V loading time issue (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/) is a good example, where GTA V did many things really well technically but then completely ignored a huge loading performance issue for many years; things like that aren't "very much a skill/experience thing", the difference is whether the company cares.

I don't think you've been following this story. If you looked into the technical details [0], and surrounding context, you would understand why your comment isn't particularly insightful. For example, terrain has nothing to do with this bug and it isn't a complex fix that interacts with "millions of lines of code." A single run of a profiler would have caught this. I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester. [0] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Casey Muratori has some great YouTube videos (a.k.a.: rants) about the end-user visible performance degradation over the last two decades despite computers becoming 10-100x faster by various metrics. E.g.: Visual Studio Rant https://www.youtube.com/watch?v=GC-0tCy4P1U How fast should an unoptimized terminal run? https://www.youtube.com/watch?v=hxM8QmyZXtg The root cause is that Microsoft simply doesn't have performance as a priority. It's not a KPI for (almost) anyone, certainly nobody in the desktop teams. Upper management doesn't prioritize input latency over telemetry or advertising "features". It's that simple. There's no pressure, no rejection of slow code, no metrics, no pride in good engineering. If it barely functions, it's good enough to ship.[1] Microsoft employees will occasionally comment to say otherwise, but their statements are contrary to easily verifiable facts, such as massive performance degradations in trivial tools such as the new Notepad, the new Terminal, and the Calculator app. The actual specific technical reasons include: - Executables used to be loaded from disk one 4KB page at a time. Execution could start as soon as the first 4KB block was loaded. For a long time now, executables are digitally signed using a hash like SHA1 or SHA256, which means that large executables can't actually start running any code until the entire file is loaded. The obvious fix -- that didn't occur to anyone at Microsoft -- is to use a Merkle tree hash instead of…

It's a shame that's their only post (https://nee.lv/)

> They may well be optimizing for a median gaming PC instead. This is how you get games that could run good if the user got more control over the model LODs, post processing effects and even render scale, but the developers/project management didn't care. What's worse, a lot of modern games have great ability to scale back and run on lower end hardware when necessary, but the companies behind them only care about that ability when it comes to getting them running on Switch or a similar constrained hardware environment, that would still let them rake in more cash. And outside of particular hacks (messing with config files, or using untrusted utilities), the users are often left powerless because a few configuration variables weren't exposed to them for whatever reason. That's actually worse than Electron apps that are typically badly optimized by default (platform overhead): it's very much like a developer in an enterprise setting choosing to go for the N+1 by looping over data in the app and doing DB calls for each iteration, yet everyone actually is okay with it. Except for the people who actually don't want their software/game to run slow, just because they cannot afford to throw unreasonable amounts of hardware resources at the problem the devs (and whoever is telling them what to do) inflicted upon them. Essentially, it's Wirth's law: https://en.wikipedia.org/wiki/Wirth%27s_law Best counterexample to this is probably e-sports titles that are optimized for stable…

This is unfairly downvoted IMO, because I think you're actually hitting on a very important point. Linux users tend to be enthusiasts, people who love software, and also primarily interact with a community where the "developer/user" is the most important figure, the one whom everything is designed around. In the Linux community, at least unless a project has a toxic developer (there are a few), bug reports are Always Good. They're how we make the software WE use better on the systems that WE use it on. Even if a report isn't fully actionable (e.g. it's a problem with graphics drivers), the report is often helpful because the bug tracker is probably public and we can try to find workarounds, or at least flag the issue for others. For closed source commercial software, especially cases where a tiny number of developers are working on the code, bug reports are Always Bad. They represent more work, work that you don't want to have to do, because at the end of the day these are people who already bought the game. You've gotten as much out of them as you're going to get out of them. If they're more trouble than they're worth (someone else in this thread claimed 90% of bug reports out of 1% of purchases), then it's obvious you should just ignore them or not port your game to their platform at all. You'd think this attitude would be different for issues that affect a lot of people: a good bug report can help you fix widespread problems that are hurting your players, but actually…

> Repairability? Other concerns, things that change the day-to-day use of the device, should weigh more. I don't think repariability is something you need to put effort into build into a product, it's something you put effort into taking out of a product. Apple's engineers definitely have things like what Louis uses for his repairs: board views, schematics, diagnostics software. Louis gets these things from illegal vendors in other countries who obtain them from Apple workers. If Apple just sold these things to him he'd be willing to pay but that's not in Apple's best interest. Also, Apple prevents Louis from buying replacement parts because they make their vendors agree to never sell the components they use to anyone but apple. > Other concerns, things that change the day-to-day use of the device, should weigh more. This avoids another massive externality: recycling and the environment. Right now Apple's recycling program takes old mostly working systems and shreds them. There's a reason we should: Reduce, Reuse, and Recycle in that order. > So maybe only these component failures are left. Reverting to the old methods of replaceable components would likely re-introduce a massive class of problems that have by and large been eliminated. The macbook repair community was at one point fixing a very common issue by replacing a component with a slightly higher spec part. The difference was 1/1000 th of a cent to 2/1000 th of a cent in cost or something like that. Multiple…

Reminds me of an article[1] that described how he cut GTA Online loading times by 70% because strlen was getting called for effectively every character in a string [1]: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

This is not quite an accurate telling of rockstar's reaction, there were actually receptive to it and paid out $10k for the discovery. Though it's an understandable mistake given rockstar's hostile history with the gta modding scene. See the original post and discussion for the whole story: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ https://news.ycombinator.com/item?id=26296339

>> Peak memory consumption is 1.3 MB. At this point you might want to stop reading and make a guess on how much memory a native code version of the same functionality would use. At this point I'd make two observations: - how big is the text file? I bet it's a megabyte, isn't it? Because the "naive" way to do it is to read the whole thing into memory. - all these numbers are way too small to make meaningful distinctions. Come back when you have a gigabyte. It gets more interesting when the file doesn't fit into RAM at all. The state of the art here is : https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ , wherein our hero finds the terrible combination of putting the whole file in a single string and then running strlen() on it for every character.

There was also the GTA wasting minutes to load/parse JSON files at startup. https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ And Skylines rendering teeth on models miles away https://www.reddit.com/r/CitiesSkylines/comments/17gfq13/the_game_does… Sometimes the performance is really ignored.

Not sure GTA 5 is the right example to list here. Remember https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/. At least for a while they didn't optimize at all.

You are missing that each update takes AGES while it tortures your disk for patching the files (on my machine it takes 15min or so, and that's on an SSD). So I agree that this is careless and reminds me of the GTA5 startup time that was fixed by a dedicated player who finally had enough and reverse engineered the problem (see https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/). I still find these things hard to accept.

Feast your eyes: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ There were HN discussions at the time as well.

funnily enough, there was an infamous bug in GTA5 for a long time that was related to using JSON : https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

JSON didn't save them: This is the same studio that handrolled a JSON parser with accidentally quadratic time complexity, making most players wait 3 to 10 minutes to load GTA Online, for 7 years, until a player got tired and found the root cause. https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

The GTA performance bug didn't get fixed because back when they were trying to make the game performant, there was no problem. They kept adding new stuff to the online part of the game, and the file being parsed grew to be ridiculously large. The person who fixed the bug mentioned "63k item entries": https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ The economics of these games requires that you remove most of the development staff after launch. The games cost hundreds of millions to develop. Everyone went off to go work on GTA6 or some other title.

This? https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ LOL

many are cpu bound for stupid things like https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ or various accidentally quadratic functions of course they shouldnt, but such is life

With a bit more digging, but very similar https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Unless you're Rockstar North: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ https://news.ycombinator.com/item?id=26296339

The link to HN was added to the blog post, itself: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/#Sm…

> I remember reading a SO question which asks for a C library to parse JSON. A comment was like - C developers won't use a library for JSON, they will write one themselves. > I don't know how "true" that comment is Either way it's a good way to get a pair of quadratic loops in your program: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

Except if it's this article - https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/