Ulrich Drepper

The best introduction is probably the original presentation paper from 2002. Fuss, Futexes, and Furwoks: Fast Userlevel Locking in Linux https://www.kernel.org/doc/ols/2002/ols2002-pages-479-495.pdf Another great paper that came after some additional experience by glibc developers: Futexes are Tricky, by Ulrich Drepper https://www.akkadia.org/drepper/futex.pdf

It is best practice and I wish it was common practice, but there's a lot of friction in the way and some people will actively work against you. Static linking glibc is painful because the former maintainer has some strong opinions: https://www.akkadia.org/drepper/no_static_linking.html Most distros force you to dynamically link every dependency if you want them to package it. So the default build for most projects is dynamic You've stumbled on a holy war between distros and guys like you, me, and Linus Torvalds[1] that want to deploy a binary and just have it work everywhere. 1: https://youtu.be/5PmHRSeA2c8?t=295 (Highly recommend watching Linus's answer ~6 minutes in)

https://www.akkadia.org/drepper/tls.pdf is also a great write-up

I've seen binary trees used for collision resolution in hash buckets in quite a few production systems. A pathological case of O(log n) is vastly better than a pathological case of O(n), if you care about it. It might be instructive for programmers using higher level languages to learn about the cache hierarchy and the hardware prefetcher in modern systems. For those interested, I'd recommend reading Ulrich Drepper's classic: What Every Programmer Should Know About Memory[1]. [1]: https://www.akkadia.org/drepper/cpumemory.pdf

His whitepaper on memory is a fantastic read, though (if with a somewhat clickbaity title). https://www.akkadia.org/drepper/cpumemory.pdf

Probably because it isn't officially sanctioned, which apparently was the reason the more common libcs never added support for bcrypt directly. - "The security departments of some customers look at recommendations they get and evaluate the deployed systems based on this. In a few places this led to the problem that the NIST warns people indiscriminately about the use of MD5..." (Note, this was from 2007 when md5 would be in roughly the same state that sha1 is now in.) http://www.akkadia.org/drepper/sha-crypt.html https://access.redhat.com/articles/1519843#ok-ok-but-why-not-bcrypt-5

Obligatory link to Drepper's classics: https://akkadia.org/drepper/no_static_linking.html https://www.akkadia.org/drepper/dsohowto.pdf Seems like the Plan 9 guys haven't heard of these? If done properly, dynamic linking is almost always better than static -- key word here is "properly", which doesn't always/often happen.

You can write mostly C code. Each processor will have different quirks. https://en.wikipedia.org/wiki/CPU_cache https://www.akkadia.org/drepper/cpumemory.pdf These are great places to start. Especially Ulrich Drepper's PDF on memory. edit: even simple things like looping can require knowledge of "bits" and "memory layout" and "CPU cache" to write the fastest code. Most people don't need to write the fastest code though.

This is the paper people usually use to explain some of the performance benefits you could reap, besides the talk about row vs column based performance: https://www.akkadia.org/drepper/cpumemory.pdf Note that I wouldn't really recommend it unless you're really going to do low level programming for high performance. It's super long, explains all the details, if you've got a basic CS college education you should know most of it already anyway. If you're just a web developer this won't actually help you as most of this improvements have already been done in the parts that matter (i.e. your database, your operating system, your interpreter and perhaps your application server).

PDF version of said same article. https://www.akkadia.org/drepper/cpumemory.pdf Need to supplement this excellent article with Row Hammer information. It doesn't cover clflush. I sent the author a note.

So this document surely belongs here: What Every Programmer Should Know About Memory https://www.akkadia.org/drepper/cpumemory.pdf You also get an answer for the question asked by the headline of this thread - in great detail (most people will probably skip a lot of details).

> This is ... the kind of content that i think should be put in a single "fact sheet". Things like that have been done before. [0] It seems to be difficult to get good Google juice. Perhaps your writeup will be the one that cuts through all of the SEOed clickbait and is the go-to guide for good IPv6 info. :) However, don't let me discourage you from taking on the task! It's sure to be edifying... or -at least- something productive to do to while away that obscenely long plane ride. Best of luck! [0] http://www.akkadia.org/drepper/userapi-ipv6.html

This can be done now with hyper-theads. It is discussed in Ulrich Drepper's excellent 'What every programmer should know about memory' in section 6.3.4. http://www.akkadia.org/drepper/cpumemory.pdf To be clear I am not declaring 'been done before, not interested' I like this blog post and if your software is cpu bound then you are almost certain memory access bound. So there is a lot of value in finding better ways to speed up access.

The best I could find is http://www.akkadia.org/drepper/tls.pdf.

The classic: http://www.akkadia.org/drepper/cpumemory.pdf Look at 6.3.2 for software prefetching. Like the rest of the paper it's very x86 centric but most of the concepts apply to other architectures.

For a way more detailed look at memory architectures and implementation, check out Ulrich Drepper's classic paper "What Every Programmer Should Know About Memory"[1] [1] http://www.akkadia.org/drepper/cpumemory.pdf

Try Drepper's (of GNU library fame) What every programmer should know about memory. http://www.akkadia.org/drepper/cpumemory.pdf Slightly dated but still excellent introduction.

The large pages feature (huge pages) still works the same way but the transparent huge pages patch has been integrated recently: http://www.slideshare.net/raghusiddarth/transparent-hugepages-in-rhel-… I personally wouldn't mess with the VM settings the way the author suggests. I don't remember doing that eight years ago anyway even with less RAM on a desktop. On a server you want the working set to fit in RAM anyway and the page cache is already pretty smart. The best resource is a general kernel book: http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/06723294… in terms of the interaction between memory, the cpu, and virtual memory What every Programmer should know about memory should be required reading: http://www.akkadia.org/drepper/cpumemory.pdf

Yea. All this and more is covered in the fantastic paper by Ulrich Drepper: What Every Programmer Should Know About Memory[1]. Was very enlightening. 1: http://www.akkadia.org/drepper/cpumemory.pdf

If you want to read something more approachable that can serve as a gateway to this, I highly recommend Jon Stokes' Inside the Machine. It goes over microprocessor architecture in a very approachable way, and contains a good chapter on the memory/cache hierarchy that is sort of the light version of a good chunk of Drepper's paper, or at least will be a big help in making sense of it. If you've never been close to the metal, or you want to catch up on newer developments, consider checking it out. Also, here's Drepper's paper compiled into a single PDF, which I think was edited a little later than the LWN publication and might have had some minor errata fixes: http://www.akkadia.org/drepper/cpumemory.pdf

snprintf_l() is not standard, but you can simulate it with uselocal() to save and restore the per-thread locale, which is standard (POSIX.1-2008) and supported by Glibc, FreeBSD, MacOS and other OSes, though Windows requires its own, different way of doing per-thread locale. It seems it was once an explicit design choice in Glibc. Here (https://sourceware.org/bugzilla/show_bug.cgi?id=10891) a Glibc ticket refers to documentation "Thread-aware Local Model, A Proposal" of the plans for the _l() functions (https://akkadia.org/drepper/tllocale.ps.gz) long ago, which are nowadays implemented in Glibc. It looks like snprintf_l() and other <stdio.h> functions were not part of that plan. That plan or something like it also made its way into POSIX.1-2008, so it seems likely that the committee gave some thought to including strftime_l() and not snprintf_l(). The paper linked above includes a rationale that tiny, potentially performance-critical functions like isalpha() need a fast version that takes a local parameter, thus isalpha_l(), because of the overhead of fetching a thread-local value inside the function. Perhaps the intent is that only those tiny functions, or even macros, whose performance would be greatly affected by the cost of fetching the thread-local locale, need a _l() version. That mostly makes sense for the functions which have _l() versions in Glibc. But with that rationale, I don't see why there is strftime_l() but not snprintf_p().

There are "stream" instructions available on some CPU. These instructions tell the CPU to store data, but not to load it into a cache line (it might actually load it into a temporary cache line that is not part of the normal cache lines, depending on architecture). This is useful is you are writing data to memory but don't read it - there's no point in loading that data to the cache line since you will not access it again. One such instruction can be used with the _mm_stream_si128 intrinsic: void _mm_stream_si128(__m128i *p, __m128i a); Stores the data in a to the address p without polluting the caches. If the cache line containing address p is already in the cache, the cache will be updated. Address p must be 16 byte aligned. Learned about it in "What every programmer should know about memory": https://www.akkadia.org/drepper/cpumemory.pdf It's from 2007, but still relevant and a good read.

I think this may be just how most programming (and perhaps everything else besides science experiments) has to be done. If you're writing assembly code, well, computers have some pretty solid low level abstractions, but do you really understand how many clock cycles that instruction there is going to take? What about all the error cases? If you're writing for the web, it's clear you can't even try to really understand how everything works in all cases across all machines. Few programmers really understand how computer memory works, but if everyone were to read this guide (http://www.akkadia.org/drepper/cpumemory.pdf) and really learn it throughly, the net benefit probably wouldn't be that great vs other things they could do with that time. It's great fun to track down every last misunderstanding or bug, but it's probably not worth it over the long term. The benefit of learning the internal details of a reliable subsystem is often not worth it and it's easier to just use it and get on with the project. That being said, when I do go to fix a bug, I try to get to the bottom of why it happens and not just stop at the first change that works. This is useful for a number of reasons: 1) If you made the bug, you probably have some misunderstanding of the system that will be fixed when you understand the bug. 2) If you're spending your time fixing an actual bug, then the misunderstanding empirically matters because this mistake has been made and actually affects someone. 3)…

This is missing out on considerations re. tls-model=local-dynamic and visibility; making variables non-preemptible by declaring them static or visibility("hidden"/"internal") significantly reduces TLS cost. https://www.akkadia.org/drepper/tls.pdf is the go-to reference for TLS models, cf. first paragraph of each subsection of section 4.

I think this, and all the other weirdness, is because usage of thread-local variables is pretty low. Probably somebody wanted to have "lightweight" threads doing work as quickly as possible accessing only local variables. Edit: ah, see https://www.akkadia.org/drepper/tls.pdf which I was clued into by https://news.ycombinator.com/item?id=43079061 The situation is much more complicated than it sounds, because the question of where the variables are isn't known at compile time. It has to be done at link time. Which may include dynamic link time as a library is loaded. That all sounds fairly horrible. All to handle static constructors. If you don't have thread-static variables, or they don't have constructors, it's much simpler.

The aversion to static linkage predates that, back many years static linkage also had other issues that have been solved today (ALSR ..). Many of the very basic points that Ulrich Drepper made some odd 20 years back (https://akkadia.org/drepper/no_static_linking.html) still hold.

Refer to 'futexes are tricky', which shows how to build a mutex with a futex—<https://www.akkadia.org/drepper/futex.pdf>—windows futexes are named WaitOnAddress, WakeByAddressSingle, and WakeByAddressAll. (That said, I think I heard somewhere that windows mutexes are decent now?)

From the other side of it, there's "How to Write Shared Libraries": https://akkadia.org/drepper/dsohowto.pdf Edit: LD_PRELOAD would have been useful to mention in the article. It's a useful mechanism, e.g. for profiling by hooking relevant functions, like the PMPI mechanism in the MPI standard.

> [H]ow does glibc maintain ABI compatibility? Alias attributes? Symbol versioning[1,2]. Kind of like aliases, but more awkward. In theory it’s not specific to glibc, in practice hardly anyone else bothers being so careful about their ABI. Note that dynamically linked musl, which Alpine uses (I think..?), doesn’t understand symbol versioning, though I expect this will only last until the first ABI break in musl. [1]: https://www.akkadia.org/drepper/symbol-versioning [2]: https://maskray.me/blog/2020-11-26-all-about-symbol-versioning

Very interesting. There was also the old Drepper paper, which was very good but this seems more accessible. [1] [1] https://akkadia.org/drepper/dsohowto.pdf