Rich Felker

> musl libc I'm conflicted on Musl. I like the idea. I like the focus on robustness. I like Felker's style [1]. But for God's sake, Musl refuses to implement basic operating system primitives we've had for decades, like dlclose(3), all because the author, from his necessarily limited perspective, just can't understand why someone would want those facilities. I'm sticking with glibc until the Musl people decide they want robustness and feature parity. By the way: PAM isn't "add in the bloat of additional backends". Dynamic loading of backends is the opposite of bloat. There's absolutely no reason Felker can't greatly improve libc robustness and maintain at least source-level compatibility with a libc we've used for decades and that has many valuable features. Musl's sin is being gratuitously opinionated. [1] http://ewontfix.com/, good for the most part. Wrong about 32-bit PIC being terrible. (What we really need is -fvisibility=hidden and -Bsymbolic by default.)

Well yes and no. I like the good parts of SystemD (not the parts this video is talking about). For me there are 3 things that SystemD gets right: - no shell scripts for service configurations - no forking services (no need for daemon mode anymore) - logging and service management is tightly coupled (this is arguably good with some bad parts, required CPU for logging with systemd is not great) And the rest is just pure madness. - DNS (but whyt?) - NTP (but why????) - system limits (seriously, da f???) - pid 1 should be a very simple service (More: https://ewontfix.com/14/) I have spent 25 years on Unix, so I guess I understand the concepts.

http://ewontfix.com/14/ is one of the classic posts here, but it deserves a good rebuttal, not the straw-man one included. (The starting point I'd make is, if init were really as simple as purported, why isn't it built-in to the kernel? Why not simply skip all the kernel code to reparent children to init, and have children die immediately instead of becoming zombies if their parent has exited? Why not remove all the code to panic if pid 1 dies, and make it just a process like any other?)

One of musl libc guys wrote quite convincing article about NULL: http://ewontfix.com/11/ There was also discussion on musl mailing list (I don't know is this best link to it): http://www.openwall.com/lists/musl/2013/01/09/1

This is a mostly-untrue statement because it makes unreliable assumptions about the host system. It depends on the vm.overcommit_memory setting and the programmer should never make assumptions about why or when malloc might fail. Read more on Rich Felker's excellent blog post here: http://ewontfix.com/3/

> In general, I have found that things which are difficult on Linux are so because the problem is difficult [...]. Hard disagree. Audio mixing is not difficult[1]. The Linux kernel guys were right - it does not belong in the kernel. The userspace story however, has been a complete shitshow for decades. I think Pipewire mostly fixed that? Not sure, sometimes I still have to log out and back in to fix audio. The funniest part? It's been working in the BSDs all along. I recommend reading the source of sndiod[1]. [1]: <https://cvsweb.openbsd.org/src/usr.bin/sndiod/> What's even worse? Probably systemd. I try not to hold a strong opinion - I tolerate it, the way I tolerate traffic when taking a walk. The technical issue however is several orders of magnitude simpler - again, the BSDs come to mind, but you can also write a complete PID1 program in about 20 lines of straightforward C[2]. I don't mind the commands being unfamiliar (they're already all different in almost every OS family); it's that the entire package is dreadfully large in scope, opaque, and I find it more difficult to interact with than anything else in this landscape. [2]: <https://ewontfix.com/14/>

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway. * Blogs with a math focus: http://blog.sigfpe.com http://bit-player.org http://mathr.co.uk/blog/ https://www.johndcook.com/blog/ * Blogs with a programming languages focus: https://existentialtype.wordpress.com https://bluishcoder.co.nz http://blog.ezyang.com http://akkartik.name http://lambda-the-ultimate.org http://mrale.ph * Blogs with a UI design focus: http://www.brandonwalkin.com/blog/ http://ignorethecode.net/blog/ http://interuserface.net * Blogs with a graphics / 3D rendering focus: http://c0de517e.blogspot.com http://diaryofagraphicsprogrammer.blogspot.com https://directtovideo.wordpress.com http://graphicrants.blogspot.com http://www.reedbeta.com http://www.realtimerendering.com/blog/ https://blog.demofox.org http://eelpi.gotdns.org/blog.wiki.html * Other interesting / more general programming blogs (many of these have a "systems" focus): http://danluu.com http://ewontfix.com http://chneukirchen.org/blog/ http://neverworkintheory.org http://apenwarr.ca/log/ https://www.snellman.net/blog/ https://www.pvk.ca http://psy-lob-saw.blogspot.com https://randomascii.wordpress.com http://fastcompression.blogspot.com https://fgiesen.wordpress.com http://blog.plover.com http://chneukirchen.org/trivium/ http://rachelbythebay.com/w/

I have been using Musl for various purposes for quite some time. I think my favourite things are: * The team are very helpful, and fix things fast and correctly. You can see some of the detail in this post on race conditions in glibc http://ewontfix.com/16/ * It is very standards compliant, so it is a good portability test for your code. It is a bit like using a BSD libc, except it does have Linux specific syscalls etc. * Static linking works, which it doesnt properly in glibc. If you want to be like those trendy Go people and provide simple statically linked single file binaries, for running in containers etc then you can. And they will be small: the glibc shared library is 1.8MB (not including rt, pthread etc), while a Musl statically linked copy of Lua for example is only 160k. Static linking is very useful for all sorts of things, my most recent use was cross compiling bianries to run under qemu-user for testing. * It is BSD licensed so you can use it in completely different projects however you want, eg OSv uses it http://osv.io/ * The code is small and readable. If you want to know how things work you can just take a look, quickly find the code and understand it. Just finding the code in glibc takes ages. I keep a reference copy just for this purpose. You can use the "cross compiler" musl-gcc that ships with it to build if your code is happy being cross compiled (should be, most things just work). If you have a lot of dependencies it might be easier to use a…

Are you talking about those ridiculous conspiracies[1][2] where systemd is broken by design so companies like RedHat can make more via support? [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#6959 [2]: http://ewontfix.com/14/

Cancellation is always such a big issue regardless of language or environment, and whether it's due to a timeout or some other action. In C, when using threads and thread cancellation there are a lot of subtleties involved: https://ewontfix.com/2/ Even in Haskell where there's generally a focus on well designed abstractions, asynchronous cancellation can fail due to badly written code that wants to catch all exceptions.

As the article eludes to it is not without its challenges, however. As an example see this issue: https://github.com/golang/go/issues/1435 which goes into some of the details why implementing a call for setuid() is not straightforward without the knowledge built into glibc. Also the trickiness of having efficient process fork/exec based on vfork: http://ewontfix.com/7/ and the considerations going into Go: https://go-review.googlesource.com/c/go/+/46173/

"Linux has an elegant and beautiful design when it comes to threads" - unifying threads and processes does simplify some things, but it makes it extremely hard to change a process-level property such as the effective user/group id of a process: https://ewontfix.com/17/.

There is also a description of what a minimal init should do here: http://ewontfix.com/14/

I do wonder how much effort and forks will be required to keep Gentoo, well, Gentoo going forwards, as more and more stuff either is either lumped into Systemd or depends on/assumes its presence. One can already see how you at least have to keep the libsystemd package installed on Debian's latest, even if you use something other than systemd as the init etc, or else apt throws a tantrum. http://ewontfix.com/15/ may be the most interesting read in all this. Notice how everything but dbus and the systemd lib have various issues. And if kdbus hits the kernel, systemd is poised to position itself as the mediator between (legacy) dbus and kdbus. All in all there is some serious "one true way"-ism coming out of the systemd camp.

It may not be deprecated, but it's a rather unsafe interface: http://ewontfix.com/7/

Rich Felker, the lead developer of musl libc, uses make to generate his ewontfix.com blog. See http://ewontfix.com/Makefile

That is indeed my point. The reason: pid 1 is special on unix. Here's a pretty decent article on the subject: http://ewontfix.com/14/

This guy is wrong, for the record. The default behavior of malloc on linux involves kernel magic to decide whether to allocate the memory. Since he wasn't actually writing any data to any of his ram, it didn't fail until he ran out of address space. If he'd actually used the memory, it would have failed sooner. The other, bigger problem is his assumption that all linux installations work like this. Frequently vm.overcommit_memory is set to disable malloc calls for more than the available amount of memory, which means the oom-killer never gets inolved. He's making a ton of dangerous assumptions based on tooling around with code instead of reading the documentation on the matter (i.e. malloc(3)). malloc can fail, and you should deal with it well: http://ewontfix.com/3/

OTOH, Linux's setuid and similar interfaces are per-thread, not per-process as required by POSIX and as what most people would expect--particularly in the context of software security. glibc and musl have to emulate the POSIX behavior, and doing it correctly is extremely tricky. See https://ewontfix.com/17/ Linux's various APIs used for container frameworks also tend to be per-thread, not per-process; that can be both useful and a giant headache, depending on context. It's somewhat ironic that Go was used for Docker as Go is pretty much the worst possible language you could choose in this regard. Go deliberately and thoroughly obscures native thread and process semantics. IIRC, Docker was already well along and established before Go even introduced an API for pinning a goroutine to a machine (kernel) thread. I once ran across a comment where the author of (I think) runc lamented his choice of Go. But I haven't been able to find it again, so it's entirely possible it's a misattribution on my part.

I think the proper title is: Linux Process and Threads Don't Mix. The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!) This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse. Check out https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.