Nelson Elhage

why does the quality of your code matter? it matters if you plan on changing it often. visiting code just to improve it's quality will likely cause bugs and cost time/money to no end. focus your energy on code that's changing most frequently or growing the fastest. 0) human angle meet to determine what of these prevent sustainable operations or growth. if something becomes quadratically more painful in relation to team growth, deal with it quickly determine what things are annoyances that would save real time (getting angry at the codebase costs time) buy time from leadership; if leadership is not on-board with this, then it will always suffer 1) code review reviewing code when a company is growing is a good way for original authors to impart knowledge or suggest architectural improvements (often ones they wish they made). it also creates accountability and a discussion forum for the items that follow. 2) testing testing is like a ratchet and it can be used to build a punch-list of the most important things to fix over time. coming up with a way to test that gets you a lot for a little is key at first. here's an example for an API: (1) snapshot your database, (2) make a bunch of api calls, (3) turn those api calls into test cases (there are creative ways to do this quickly), (4) run them all the time, (5) reduce the number of failing tests over time find ways to turn your system into a function (use property based testing or fuzzing and you get a _lot_ for a…

I don't know that _efficient_ is the best word. If you use goto to force a particular control flow rather than a more constrained form of control flow (e.g. if), you make it harder for the optimiser to work; it can only make "as-if" changes, ones that mean the code that executes looks as if it's what you wrote. The most efficient control flow is one that describes only what your algorithm needs, coupled with an optimiser that can exploit the particular flow you're describing. Among the many things discovered by the author of https://blog.nelhage.com/post/cpython-tail-call/, Clang/LLVM was able to optimise the standard switch based interpreter loop as if it had been written with computed gotos.

This is a tech-focused selection from my personal list, with a few more business-y sources thrown in: - https://erikbern.com - https://lethain.com - https://commoncog.com - https://scattered-thoughts.net - https://danluu.com - https://ideolalia.com - https://staysaasy.com - https://collabfund.com - https://blog.nelhage.com - https://jacobian.org - https://equalventures.substack.com - https://defmacro.org - https://spakhm.com - https://apenwarr.ca - https://scarletink.com - https://benkuhn.net - https://cpojer.net - https://brandur.org - https://zeptonaut.com Let me know if you want some reading recommendations for any of these!

I agree entirely with you that Rust, today, is well-adapted to people who like to learn by the "fuck around and find out" route. But it often feels as if the Rust community doesn't believe that people who prefer to learn by reading accurate documentation exist. I commend the following page to you to get an idea of the mindset involved: https://blog.nelhage.com/post/computers-can-be-understood/ (No doubt it will gladden your heart to know that I updated the Reference to document the somewhat surprising ways in which integer literal expressions decay earlier this year.)

Nelson is brilliant- i've enjoyed his blog tremendously over the years. Especially his posts on how to properly think about testing. - https://blog.nelhage.com/post/testing-and-feedback-loops/

The author has a followup post about that exact case and experiences trying to understand them: https://blog.nelhage.com/post/systems-that-defy-understanding/

The author had a followup post about this that I found more interesting: https://blog.nelhage.com/post/systems-that-defy-understanding/

Marshalling bugs in other languages and frameworks: - Java: WebSphere, WebLogic, JBoss, Jenkins : http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jbos… . Admittedly most of these are through sidechannels and nothing as obvious as sessions, but it's the same mistake. - Python: https://blog.nelhage.com/2011/03/exploiting-pickle/ . Unpickling got at least Cisco Web Security Appliances: http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/C… - PHP : Of course, it's PHP and of course, it's WordPress . https://vagosec.org/2013/09/wordpress-php-object-injection/ It's hard to attribute malice to an obvious mistake that everyone makes.

I put a bit further thought into this. The claim is that because the pty has user permissions its possible to hijack it, but that really hasn't been true for years which is why the two PoC do things like use reptyr (involves replacing fd's with ptrace [1], needing either a parent relationship or capabilities that aren't default) or running a command like netcat directly from the pty you want to control [2] (essentially ttyjack [3]). If you look at the implementation of TIOCSTI [4], 99% sure this is what the new PoC is doing, you can see why you can't do this across ttys. This goes back ages, even into some 2.6 kernels it seems before I got bored going back kernel versions. I went and tested all this on a Ubuntu 22.04 box, not possible unless there is a new trick to hijack ttys. I tried the TIOCLINUX across ttys physically at the machine and that didn't work either. [1] https://blog.nelhage.com/2014/08/new-reptyr-feature-tty-stealing/ [2] https://twitter.com/hackerfantastic/status/1786080968958161229 [3] https://github.com/jwilk/ttyjack [4] https://elixir.bootlin.com/linux/latest/source/drivers/tty/tty_io.c#L2…

> Unless there is clear evidences upfront that the project will be a piece of software where local performance is highly critical, it makes sense to favor code readability and maintainability over optimality. Sure, but let's also not confuse "optimal" with "reasonable". One of the major challenge of modern programs is how slow they are at every level. Very often, this bad performance can be attributed to a style of programming that tries to completely ignore that the program will run on a real machine with real hardware. A little bit of mechanical sympathy (e.g., operations that make good use of the CPU caches or don't confuse the branch predictor) can yield a program that is 10x faster than a naive implementation, with little to no loss of readability or maintainability. (In fact, as noted by Nelson Elhage [1], faster programs enable simpler architectures, which helps make them more readable and maintainable.) In FP languages, programmers face an extra difficulty: the distance between the code they write and the machine code that will be executed is greater than in languages like Rust or Go. They will need to be knowledgeable about the language, its libraries, and its compiler to avoid the pitfalls that could make their programs slower than would be reasonable (e.g., avoiding unnecessary thunking in Haskell). [1] https://blog.nelhage.com/post/reflections-on-performance/#performant-f…

I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code. For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4]. [1] https://www.oreilly.com/library/view/natural-language-processing/97810… [2] https://github.com/karpathy/minGPT [3] https://jalammar.github.io/illustrated-transformer/ [4] https://blog.nelhage.com/post/transformers-for-software-engineers/

> It is recommended to learn if subtransactions are used in your systems. If they are, it does not immediately mean that they need to be eliminated – it all depends on the risks of your particular case. Evaluating such risks is a complex task that requires deep Postgres expertise, so if you need such an analysis, consider hiring a PostgreSQL expert. It is quite disheartening that this essentially echoes this quote from the related blog post https://buttondown.email/nelhage/archive/22ab771c-25b4-4cd9-b316-31a86… > It is entirely possible to operate Postgres safely and with high performance and throughput at scale … but essentially the only way to do it is to have ready access to deep PostgreSQL experience and expertise on your team; people who Just Know where the landmines are because they’ve seen them before. And https://blog.nelhage.com/post/some-opinionated-sql-takes/: > My personal choice: MySQL > The devil is in the details, but for me, as a default, and despite all these pet peeves, I would start with MySQL. > As for Postgres, I have enormous respect for it and its engineering and capabilities, but, for me, it’s just too damn operationally scary. In my experience it’s much worse than MySQL for operational footguns and performance cliffs, where using it slightly wrong can utterly tank your performance or availability. In addition, because MySQL is, in my experience, more widely deployed, it’s easier to find and hire engineers with experience deploying and…

Another recent Postgres-complaint post from one of the best engineers I've worked with: https://blog.nelhage.com/post/some-opinionated-sql-takes/ Quoting his conclusion: > As for Postgres, I have enormous respect for it and its engineering and capabilities, but, for me, it’s just too damn operationally scary. In my experience it’s much worse than MySQL for operational footguns and performance cliffs, where using it slightly wrong can utterly tank your performance or availability. … Postgres is a fine choice, especially if you already have expertise using it on your team, but I’ve personally been burned too many times. He wrote that shortly after chasing down a gnarly bug caused by an obscure Django/Postgres crossover: https://buttondown.email/nelhage/archive/22ab771c-25b4-4cd9-b316-31a86… Personally, I'd still opt for Postgres every time – the featureset is incredible, and while it may have scary footguns, it's better to have footguns than bugs – at least you can do something about them. Still, I absolutely wish the official Postgres docs did a better job outlining How Things Can Go Wrong, both in general and on the docs page for each given feature.

In posts talking about how someone made something fast, I see the same idea repeat time after time: it has linear time complexity, has great cache locality, and saturates all available processors. It's annoying to me how much time I had to spend outside of school before I realized that this is the real recipe for fast code. In particular, even code that is nominally O(n^2) by standard measures is still a dis-economy of scale: the bigger the problem the slower it gets. In school, there's a premium given for finding a polynomial-time solution, but in practice every job where I've been tasked with making something fast, only linear time complexity was good enough. In practice these linear algorithms tend to also be completely data parallel and have great cache locality. Not only do these properties make the code fast the second you harness them, but then tend to be exceedingly simple architectures to maintain over time. For some other reading I really like on software performance in the real world: - https://blog.nelhage.com/post/why-sorbet-is-fast/ - https://blog.nelhage.com/post/reflections-on-performance/

See: https://blog.nelhage.com/2010/10/amd64-and-va_arg ("To start, any function that is known to use va_start is required to, at the start of the function, save all registers that may have been used to pass arguments onto the stack, into the 'register save area', for future access by va_start and va_arg. This is an obvious step, and I believe pretty standard on any platform with a register calling convention. The registers are saved as integer registers followed by floating point registers. As an optimization, during a function call, %rax is required to hold the number of SSE registers used to hold arguments, to allow a varargs caller to avoid touching the FPU at all if there are no floating point arguments.") When va_start is used, it needs to save argument registers to the stack in the prologue of the function. The program is free to use different conventions (format string, sentinel value) to signal to the callee how many arguments there are. But the code generated for va_start has no way of knowing what convention the program happens to use.

This article helped me a bit when porting Shake to x86_64 (the first in the series is linked from the article): http://blogs.msdn.com/b/oldnewthing/archive/2004/01/14/58579.aspx The x86_64 calling convention can be a bit hairy. C.f. http://blog.nelhage.com/2010/10/amd64-and-va_arg/ The best thing about really large C++ applications with assembly glue deep in their core is that nothing works until you rewrite the glue for the new platform. You can't debug the thing like a normal app when its hand-rolled threading and mutex library from 1997 doesn't work. So you tread forward; only suckers use debuggers. You can tell what it's supposed to do by reading the assembly and function names from the other platforms... (ppc, alpha (SGI), and i386 in this case). Naturally, none of the assembly parts had any documentation. Honestly, it's probably good that they didn't (j/k). My guess as to what they were thinking: "if they don't understand it then they shouldn't be writing the asm for the next platform". I guess they were right. Finally, you get the glue right and it's an "ah-ha" moment. Everything suddenly starts working.

I'm not sure of anything. I agree that priv escalation regularly happens, I was around for cve-2010-4258, before they got all branded and named. GP was asking why Cloudflare should be so afraid that they wrote that post and I'm saying their fear is reasonable. https://blog.nelhage.com/2010/12/cve-2010-4258-from-dos-to-privesc/

This is interestingly analogous to nelhage's memory sampling approach https://blog.nelhage.com/2013/03/tracking-an-eventmachine-leak/#search… where if something is leaking "at scale", let it - then gcore it, and pick a random page of the core and see what's there. (An even cheaper variant I've used a few times with python is to just run "strings | sort | uniq -c | sort -n" on the core, so you get the most common strings which are usually part of object metadata and give you a solid hint without needing to go in with a debugger...) The ^C approach basically samples time instead of memory.

If you like this tool, you might also be interested in reptyr, which lets you reparent a process to a different tty. https://blog.nelhage.com/2011/02/changing-ctty/

https://blog.nelhage.com/post/regex-crosswords-z3/ was posted a few days ago here and is an interesting way to solve these

Good to know! Thanks for confirming. Yes, I would guess that the tail call interpreter explains part of the difference between 3.13 and 3.14. Previously the overall improvement to the interpreter has been measured at 1-5%, or even 10-15% depending on the compiler version you are using: https://blog.nelhage.com/post/cpython-tail-call/ If your benchmark setup is easy to re-run, it would be awesome to see numbers that compare the tail call interpreter to the build where it is disabled, to isolate how much improvement is due to that.

As an alternative to the computed gotos, you can use switch/case in Clang or GCC to achieve basically the same thing. :-) It becomes a jump table in most cases. (The article claims that a jump table gives smaller code and fewer branch misses, but it doesn't actually give any numbers, and enough things in there are dubious enough that I'm not convinced they ever measured.) https://blog.nelhage.com/post/cpython-tail-call/ has made the rounds a lot recently, and explores this for Python's bytecode interpreter.

I was curious how this would look in C, and I found this article[1] how this could look in C, apparently with very little overhead. And as I just saw, Python 3.10 also introduced a NewType[2] wrapper. I'll have to see how that feels to handle. 1: https://blog.nelhage.com/2010/10/using-haskells-newtype-in-c/ 2: https://typing.python.org/en/latest/spec/aliases.html#newtype

> C gives pointers one past the end of an array meaningful semantics Nelson Elhage wrote about an alternate interpretation: https://blog.nelhage.com/2015/08/indices-point-between-elements/

A common metric for how much actual content has changed is the Jaccard Index. Even for large numbers of datasets that are too large to fit in memory it can be approximated with various forms of MinHash algorithms. Some write up here: https://blog.nelhage.com/post/fuzzy-dedup/ https://en.wikipedia.org/wiki/Jaccard_index

"Efficiency trades off against resiliency" https://blog.nelhage.com/post/efficiency-vs-resiliency/

> nothing about computers is magic In this vein I think about nelhage's "Computers Can Be Understood" from time to time - https://blog.nelhage.com/post/computers-can-be-understood/ .

You might be thinking of: https://blog.nelhage.com/post/systems-at-capacity/

This is evocative of the efficiency against resilience balance mentioned in another front page article today: https://blog.nelhage.com/post/efficiency-vs-resiliency/ https://news.ycombinator.com/item?id=35586607 With this metaphor collections of long appreciated arts become foundational while the fringe which may develop new aesthetics efficiently meets the needs of the cultural fringe while not being resilient as long term broadly appreciated artistic works.

I agree that the x32 ABI is probably a technological dead end, but I don't agree with "just" use a custom heap. I tried in C++ and it's not easy. You lose a lot of type safety and you have to basically rewrite every line of your program. I mentioned the Zig compiler here: https://news.ycombinator.com/item?id=33797368 Another program I remember that uses small integers instead of pointers is the Sorbet type checker (not coincidentally it also has a fine-grained AST code representation): https://blog.nelhage.com/post/why-sorbet-is-fast/ I peeked at the code awhile back, and it's not really idiomatic C++. It's obviously doable but you have to architect the program from the start for it -- it's not feasible to retrofit this technique onto existing software.