Joyful Bikeshedding

I also wrote about distributed locking a while ago, particularly about implementing one on Google Cloud Storage. https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-distributed-… This is useful because cloud storage is very cheap and serverless. It certainly beats running a Redis or PostgreSQL instance. In my research and implementation I took care of the problems described in this article (as well as the problems I encountered in other distributed lock implementations). Clients either freezing or outright crashing or disappearing is definitely a major problem. So timeouts are a must. But you can't just have a timeout because you don't know whether a client is arbitrarily delayed or whether it has disappeared. Furthermore, some workloads have an unknown completion time so you can't pick any sensible timeout. I address this by encouraging clients to behave as follows: 1. Make the timeout relatively short, but regularly refresh your lease to let others know that you're still alive. 2. Your lease may at any time be taken over by another client that thinks you've disappeared. So you must regularly check whether your lease on the lock is still valid. This checking must be done way, way before your timeout expires, in order to account for the possibility that you can be arbitarily delayed at any moment (GC pauses, scheduling pauses, network delays, etc). I recommend checking in an interval at most 1/8 of the timeout.…

It’s definitely surprising that macOS does this, but it makes me also wonder if the authors considered just modifying the binary so that the dynamic linker could find the library: https://www.joyfulbikeshedding.com/blog/2021-01-13-alternative-to-maco…

Not sure if this is jargon-free enough, but maybe it helps: https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic…

One thing that has been known for a while is that there are allocation patterns where LibC malloc doesn't give back a sufficient amount of pages even when the pages are clean. See for example https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-me… . In the end I think the consensus was that jemalloc was just better than invocing malloc_trim, but invoking malloc_trim now and then can certainly be a lot better than using neither malloc_trim or jemalloc.

https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-me… is a pretty interesting resource about this.

> At this point jemalloc is so good Some folks disagree: > [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy. via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-status-of-ruby-…

> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies. Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1]. See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic… for a recent basic ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html [0] http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html [1] that's Brendan Gregg saying it: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html — having contributed to it and being one of the leading dtrace experts