Evan Jones

Comparing fork to gets is perhaps a bit unfair to fork. I was being lazy when I tried to suggest a spectrum of deservedly rare api calls. :) It’s not broken like gets or outdated like sbrk, but it doesn’t compose well with other parts of the system and there are enough caveats that I recommend more use-case targeted alternatives wherever possible. For example: http://www.evanjones.ca/fork-is-dangerous.html

I plan to cover CPython's memory management in the future posts. In a nutshell, an object gets destroyed when its reference count hits 0. In this case, CPython calls `tp_dealloc` [1] of the object's type. The `tp_dealloc` slot releases all the resources the object owns and frees the memory. The implementation of `tp_dealloc` differs for different types. Eventually, the `free` function of the memory allocator is called to free the memory. A memory allocator is a set of functions to manage memory. The default memory allocator for objects is pymalloc. It allocates small objects (<= 512 bytes) using the arena allocator [2] and falls back to the raw memory allocator otherwise. The latter calls the `free()` library function to free the memory. The Python/C Reference Manual has a great section on memory management [3]. [1] https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_dealloc [2] https://www.evanjones.ca/memoryallocator/ [3] https://docs.python.org/3/c-api/memory.html

> I really don't get what people mean when they say they're against frameworks. Your app that you're making with no framework...how are things structured? Do you just repeat yourself everywhere? So, there are frameworks and there are libraries. With a framework, you use the structure provided by the framework, and hook pieces of your code into it; with a library, you structure your code to your problem, and use libraries to provide functionality. Frameworks are generally easier to get started with: they provide a lot, and their pieces generally work well together. But inevitably, the day will come when you need to do something not envisioned by your framework, and then you're stuck: it can be arbitrarily difficult, painful and/or low-performance to do what you need. That last point is also worth mentioning: because a framework tries to be good at providing the same structure for a lot of different problems, it inevitably is a jack of all trades and a master of none. This can end up meaning poor performance, due to needing to call lots of functions to do simple things (imagine some hellacious Java framework with AbstractFactoryVisitorSingletons), or it can be a pain on the programmer, or both. Meanwhile, libraries are just that…libraries of code that the programmer can use to implement needed functionality. How he stitches them together is up to him. With a framework, your code lives within someone else's structure; using libraries, other people's code is called by…

It's a long standing bug, setenv and unsetenv are not thread-safe https://www.evanjones.ca/setenv-is-not-thread-safe.html

Previously on setenv being a terrible thing: https://www.evanjones.ca/setenv-is-not-thread-safe.html (discussion: https://news.ycombinator.com/item?id=38342642 first comment is even about it causing issues in Rust)

It's time to move beyond this attitude and make things safe by default. For example, Solaris has a safer version of setenv(). "It is ridiculous that this has been a known problem for so long. It has wasted thousands of hours of people's time, either debugging the problems, or debating what to do about it. We know how to fix the problem." https://www.evanjones.ca/setenv-is-not-thread-safe.html

Perhaps related: "Setenv Is Not Thread Safe and C Doesn't Want to Fix It" (2023) -- https://www.evanjones.ca/setenv-is-not-thread-safe.html and HN comments regarding how rust was affected: https://news.ycombinator.com/item?id=38342642 Decades old footguns - aaargh!

Galactica is useless. It can only generate answers. (Pablo Picasso, see https://www.evanjones.ca/picasso.html)

TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.

The blog post announcing this might be a better read: https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a6058… (And while reading that and thinking "I swear I recently read something else talking about applying TCP congestion control to RPC queueing". And indeed I had: http://www.evanjones.ca/prevent-server-overload.html)

You need ECC /and/ pervasive checksumming. There are too many stages of processing where errors can occur. For example, disk controllers or networks. The TCP checksum is a bit of a joke at 16 bits (it will fail to detect 1 in 65000 errors), and even the Ethernet CRC can fail - you need end to end checksums. http://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html

People deal with memory leaks in Java on daily basis, eg: https://medium.com/@milan.mimica/everybody-leaks-f210631f13ef http://www.evanjones.ca/java-bytebuffer-leak.html http://www.evanjones.ca/java-native-leak-bug.html

I can see where my statement was ambiguous, I'd intended to say that the GC developers spend tons of time tuning their implementations to fit their users' workloads. I will say that "government project which serves the whole country now" doesn't really specify how many requests you're doing or what kind of pauses are acceptable. If it's a traditional website, a 300ms GC pause might be invisible. A 2s pause might even be invisible. For a service doing 30,000 requests/s (~1-2k/s per machine), a single STW pause of 2s is _very_ disruptive. Even 300ms is high. A 300ms pause is 600 queued requests @ 2k/s. We certainly tune this service to eliminate most STW GC. We were even hit by http://www.evanjones.ca/jvm-mmap-pause.html .

Anyone with enough hardware can observe this sort of problem: http://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html When The CRC and TCP Checksum Disagree: http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-9-1… Alternately, just look at "netstat -s" for any machine on the Internet talking to a bunch of others. Here's the score for the main web host of the Internet Archive Wayback Machine: 3088864840 segments received 2401058 bad segments received.

A few years ago some of the intel SSD drives has issues with this. http://www.evanjones.ca/intel-ssd-durability.html Drive was advertising FUA but it didn't actually implement it (or implemented it incorrectly). Then there were drives that were storing things in cache and reporting as written but didn't actually have capacitors to write out the data in case of power failure. http://lkcl.net/reports/ssd_analysis.html , also the paper from FAST13. I think in 2016 the situation looks much better.

Its actually very useful, although very weak. The link layer check, at least in the case of Ethernet, really only protects your data "on the wire". It doesn't protect it inside the switches. It turns out there are failure modes, such as the one that caused this issue, where the packets get corrupted inside the switch (probably due to bad RAM). Details about how this can happen: http://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html After this kernel bug was fixed, the TCP checksum did its job and discarded all the corrupt packets.

obligatory link: https://www.evanjones.ca/linear-search.html

A few links gathered from a quick google search as a primer: http://stevehanov.ca/blog/?id=97 https://www.evanjones.ca/portable-linux-binaries.html https://insanecoding.blogspot.com/2012/07/creating-portable-linux-bina… https://rpg.hamsterrepublic.com/ohrrpgce/Portable_GNU-Linux_binaries https://github.com/wheybags/glibc_version_header In other words: there are a lot of steps and a lot of gotchyas to doing this that you're glossing over. Linux userland libraries are generally designed with the intention that an army of third-party maintainers will integrate all of this desperately developed software together and place it in a repo. Naturally every distribution wants to do things a little differently too, and they have a habit of changing it up every couple years. When you try to step out of that mold, even just to static link, things unsurprisingly become more difficult. Whereas Windows, Mac, Android, etc. have been designed since the beginning not to require that sort of thing and it is consequently a much, much more straightforward process. I'm curious why, since you seem to believe the process is so straight-forward, you think it is that so few people distribute a simple binary? Why were Flatpak and AppImage invented?

I'm glad you've at least looked into this, but you have a number of things wrong: First of all, it is not a single algorithm, it is a family of algorithms This argument holds for SHA1, but all modern cryptographic hash functions are also families. The SHA3 family has SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128, and SHAKE256 For a CRC of size N, you have to also choose a N-bit polynomial, N-bit starting value, and N-bit output XOR value. There is no single standard, there are tens or hundreds of popular options Apparently you've never heard of CRC32C! http://www.evanjones.ca/crc32c.html Most programming languages have some CRC32 implementation - but do not document what parameters they use, or use different notation for the same parameters (forward and reverse), or do not let you change the parameters, or do not let you change the CRC size, or all of these. There is no easy way to get a "standard CRC64 or CRC128" compatible across platforms without putting it together from github snippets and example code yourself. Here's CRC32C implementations in a handful of popular languages: - C: https://software.intel.com/en-us/node/503522 - Java: http://download.java.net/java/jdk9/docs/api/java/util/zip/CRC32C.html - JavaScript: https://www.npmjs.com/package/fast-crc32c https://www.npmjs.com/package/crc32c - Go: https://github.com/jacobsa/gcloud/blob/master/gcs/gcsutil/crc32c.go - Python: https://github.com/ludios/pycrc32c - Ruby:…