Graham Christensen

This post has a great overview: https://grahamc.com/blog/nix-and-layered-docker-images/ tl;dr it will put one package per layer as much as possible, and compress everything else into the final layer. It uses the dependency graph to implement a reasonable heuristic for what is fine grained and what get combined.

I personally remember being inspired by Erase your Darlings and Paranoid NixOS Setup back in the day, less for the hardening measures and more because of how great the Nix syntax looked. Huge, monumental ass-pain setups could be scripted away in one or two lines like it was nothing. You could create wildly optimized configurations for your specific use-case, and then divide them into modules so they're portable. It's not advisable to switch to one of these paranoid configurations outright, but they're a great introduction to the flexibility provided by the NixOS configuration system. I'd also recommend Xe's documentation of Nix Flakes, which can be used on any UNIX-like system including macOS: https://xeiaso.net/blog/nix-flakes-1-2022-02-21/ https://grahamc.com/blog/erase-your-darlings/ https://xeiaso.net/blog/paranoid-nixos-2021-07-18/

This is linked in the footer of this page, but Graham Christensen has an excellent blog post “Erase your darlings” [1] that explains why you might want to do this. There is a script floating around on github [2] that does the install automatically, but I needs to be modified slightly to work on newer nixos versions. I have a forked version [3] that I used this morning, but my decisions might not make sense for everyone, but it’s provided as-is for now :) 1. https://grahamc.com/blog/erase-your-darlings/ 2. https://gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de3e 3. https://gist.github.com/jbott/531b9d555dae7f197f25326ef251f164/ae1478e…

I use colmena[1] to manage one nixos configuration for multiple machines: - laptop - desktop - server - rpi nas I also wipe my entire rootfs every boot with a zfs snapshot rollback[2] using the impermanence module[3] to keep specific stateful data one one of two datasets with regular snapshots: one is backed up with zfs send, the other is just for cache between reboots. It took a little puzzling to get started, because I didn’t know about the impermanence module at first, so I built my own hacky solution. But I really love this setup. And the way I don’t have cruft to clean. Also my backups are so much smaller now :’-) [1]: https://colmena.cli.rs/ [2]: https://grahamc.com/blog/erase-your-darlings/ [3]: https://github.com/nix-community/impermanence

It’s much less plug-and-play than the options in the article… but I’ve had fun with a related idea, which is running Nixos on storage where root is erased on every boot. My home server is set up this way. At the time, I more or less followed this guide, which explains things nicely: https://grahamc.com/blog/erase-your-darlings/ Been erasing root for almost 2 years I think now. Works great!

Yea and the best part was that installing NixOS was dead easy. I followed Graham Christensen's instructions[1] and had nix create a personalized image with the latest linux kernel and some other stuff. Then I just flashed and booted from that image after partitioning. Honestly it was dead simple and its so hard to go back to the ad-hoc system config style a la Arch linux and other distros. I'm probably a lost cause now because I think I'm going to convert my entire raspberry pi cluster to NixOS from ubuntu. [1]: https://grahamc.com/blog/nixos-on-framework

This is a great blog post on how to do this on nix, https://grahamc.com/blog/erase-your-darlings . My Nix machine is stateless apart from a few directories such as home that I want to remain across reboots. When I shutdown/restart the server anything not in a mount I’ve explicitly configured to be persisted, gets erased. The initial setup was a bit of work but the end result works flawlessly.

There are some technical objections to the design of flakes, but the controversy in the community is primarily social imo. The reading I'd recommend as someone sympathetic to both 'sides' isn't really a technical comparison, but this: https://grahamc.com/blog/flakes-are-an-obviously-good-thing As far as technical pros and cons, it's worth looking at colemickens' PoC demonstrating that in principle, you can use Nix to generate literally identical output with and without flakes: https://github.com/colemickens/nixos-flake-example > Both approaches it's also worth noting that prior to flakes, there was no single alternative approach to the main problems flakes aims to solve. There were (and are) a whole bunch of competing and bespoke practices for pinning nixpkgs and other Nix sources. You can take a look at niv and nix-thunk to get a sense of them if you want

If your interested in digging deeper into building docker containers with nix this is my favorite post on the topic: https://thewagner.net/blog/2021/02/25/building-container-images-with-n… Essentially you can use any nix package (including your own) to create the environment. And if you really want to understand it, Graham Christensen (the contributor of the buildLayeredImage function) wrote a really good blog post on how it works: https://grahamc.com/blog/nix-and-layered-docker-images.

> For that case, it's harder to argue that Nix is worth spending the time compared to just slowly setting up each computer manually. Actually that’s an even better justification for using NIx instead of slowly setting up each computer manually. If you’re setting up a new computer every week, or even every month, you tend to remember all the packages, configs, and tweaks you need just from frequently doing it. But if you need to set up a computer once a year, you forgot all that stuff, making it even more valuable to have it all scripted in a deterministic build. This blog post “Erase Your Darlings” discusses this issue, along with a really interesting solution to it: https://grahamc.com/blog/erase-your-darlings

Use nixos to run your k8 cluster if you are insane (or have the $$$ for salaries) enough to run your own k8. See https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/cl… Specifically "services.kubernetes.apiserver.enable = true" My strong recommendation is to use nix to generate super optimised and amazing docker images then use them with k8 on a PaaS. https://grahamc.com/blog/nixos-on-zfs If using k8 (self host or PaaS), keep an eye on https://github.com/xtruder/kubenix as it'll blow your mind. Noyaml, infrastructure testing framework, deployment etc using nix.

Like I said, it was very preliminary research... I reached the bit where pypi2nix did "nix-env -if https://github.com/garbas/pypi2nix/tarball/master" and wasn't super happy about the implications of "just use master" for production readiness. If it works, though, that's great! The more general point though is that in my experience no tool is perfect, or completely done, or without problems. E.g. the cited https://grahamc.com/blog/nix-and-layered-docker-images suggests you need to spend some time manually thinking about how to create layers for caching? Again, very preliminary research—I know people are using it, I'm just skeptical it's a magic bullet because nothing tends to be a magic bullet.

That's cool, I'll need to read up on the v3 design. I think you are right about the rebuilds, but maybe not in all cases.. If you had. FROM nodejs:whatever as js RUN npm build FROM golang:whatever as go RUN go build FROM base COPY --from js app.js COPY --from go app If only 'base' was updated rebuilding the image would just need to re-run the COPY commands. The only way everything would get rebuilt is if you also updated the go and js images. I think for more complicated applications https://grahamc.com/blog/nix-and-layered-docker-images is ultimately the best way to do things... ensuring that every step is isolated with explicit dependency information.

One should make a distinction between: * The general idea of mixing together filesystems+folders to achieve re-use/sharing/caching. * The "Dockerfile" approach to this - with its linear sequence of build-steps that map to a linear set of overlays (where each overlay depends on its predecessor). The "Dockerfile" approach is pretty brilliant in a few ways. It's very learnable. You don't need to understand much in order to get some value. It's compatible many different distribution systems (apt-get, yum, npm, et al). But although it's _compatible_ with many, I wouldn't say it's _particularly good_ for any one. Think of each distribution-system -- they all have a native cache mechanism and distribution infrastructure. For all of them, Dockerization makes the cache-efficacy worse. For decent caching, you have to apply some adhoc adaptations/compromises. (Your image-distribution infra also winds up as a duplicate of the underlying pkg-distribution infra.) Here's an alternative that should do a better job of re-use/sharing/caching. It integrates the image-builder with the package-manager: https://grahamc.com/blog/nix-and-layered-docker-images/ Of course, it trades-away the genericness of a "Dockerfile', and it no doubt required a lot of work to write. But if you compare it to the default behavior or to adhoc adaptations, this one should provide better cache-efficacy. (All this is from POV of someone doing continuous-integration. If you're a downstream user who fetches 1-4…

>"grahamc’s “Erase your darlings” blog post is an amazing example of what a snapshotting filesystems (zfs) combined with an immutable, infrastructue-as-code OS (NixOS) can achieve. To summarize the post, grahamc demonstrates how to erase the root partition at boot while opting in to state by getting NixOS to symlink stuff to a dedicated partition. This restores the machine to a clean state on every boot, preserving the “new computer smell”. I believe the main selling point of this concept of opt-in state is that it makes it dead simple to keep track of ephemeral machine state (everything not explicitly specified by your NixOS configuration) and enforces elimination of Configuration Drift. While the benefits of this are clear for servers, this also works pretty well with workstations and laptops, where you gradually accumulate junk in /etc and /var which you never can be completely confident in deleting.1" Related: https://dzone.com/articles/configuration-drift https://grahamc.com/blog/erase-your-darlings https://news.ycombinator.com/item?id=22856199

These are all good points. To be honest, I'm skeptical about the future of NixOS: it's wonderful as a user distribution, but for production servers it's a hard sell to migrate off a stable, enterprise-grade OS like CentOS, Debian, or openSUSE. And as you say there's a long-term trend toward cattle servers and lightweight throwaway containers, which do not need features like atomic rollback and deterministic versioning, because we can simply snapshot the image and do a blue-green deployment with easy rollback at the orchestration layer. That said, NixPkg (the package manager) is incredibly powerful, because it can build containers more quickly than Dockerfiles [1] or other specifications -- and it can do so in an entirely reproducible manner, with efficient caches for CI servers and developers. This becomes particularly powerful if your team works in a compiled language with long build times, like C++, or if your team works with a variety of languages. Particularly in scientific computing, it's not uncommon to have a lot of Python {2,3} with C/C++ dependencies, and perhaps a cmdline utility in another language or two. At this point something like 'pip install' no longer really cuts it, and NixPkg can step in to produce app images and development/CI environments in a snap. These images can then be backed into your favorite container runtime with the confidence that you could rebuild the entire container with a 1-line patch and actually have the same contents everywhere, and…

grahamc wrote down how to optimize this (within the constraints of max 128 layers supported by OCI) to categorize/prioritize packages by popularity/dependencies which lead to dramatic speed-ups for pulling updates or related images. https://grahamc.com/blog/nix-and-layered-docker-images/

I haven't tried it yet, but the evil twin to this practice is to nuke everything periodically to ensure that your agent isn't relying on any filesystem state that it hasn't specified builds for (i.e. https://grahamc.com/blog/erase-your-darlings/). They tend to slip out of declarative mode and start making untracked changes to the system from time to time.

Interesting, and good luck to OP. I feel a little clickbaited, cause I was hoping for a port of stateless infra (although it may be of little use on a (dev machine :p) [1] 1: https://grahamc.com/blog/erase-your-darlings/

If you were really hardcore you'd erase your entire OS whenever you're done for the day: https://grahamc.com/blog/erase-your-darlings/

> Just wondering what you'd get, if you take the above to an extreme? You might get some shade of nixos: https://grahamc.com/blog/erase-your-darlings/ (Frankly, default nixos already does a bit of the whole "apps aren't installed per se" thing - `nix shell nixpkgs#foo` gives you a shell with foo "installed" but only until you exit the shell and run garbage collection.)

It’s really not that bad. However, with a standard NixOS setup, you still have a tremendous amount of non-reproducible state, both inside user accounts and in the system. I’m running a “Erase your darlings” setup, it mostly gets rid of non-reproducible state outside my user account. It’s a bit of a pain, but then what isn’t on NixOS. https://grahamc.com/blog/erase-your-darlings/ Inside my user account, I don’t bother. I don’t like Home Manager.

Personally, my NixOS system is wiped every time it boots up [1]. So no, I have no need for a tool like sysclean. [1]: https://grahamc.com/blog/erase-your-darlings/

"It depends." poetry2nix is pretty good! Generally, exporting to an OCI image, or targeting a production environment that can run Nix closures natively is the way we go about it. Many of our services use buildLayeredImage[0] and target Fly.io. 0: https://grahamc.com/blog/nix-and-layered-docker-images/

They are likely talking about using https://github.com/nix-community/impermanence, which is inspired by ideas in https://grahamc.com/blog/erase-your-darlings/ Edit: comments on the latter at https://news.ycombinator.com/item?id=22856199

> You can easily have a bunch of generations you just think are ok. Unless you erase your darlings: https://grahamc.com/blog/erase-your-darlings

Smaller docker images for one: https://grahamc.com/blog/nix-and-layered-docker-images Nix gives repoducibility, docker gives repeatability but not reproducibility. Also see the video "Use flake.nix, not Dockerfile": https://www.youtube.com/watch?v=0uixRE8xlbY you can swim upstream and make your own docker image reproducible, but that doesn't change an ecosystem of images that aren't

https://grahamc.com/blog/erase-your-darlings did a pretty good Nix (and ZFS) sell. Of course, it was written by a guy who actually works on NixOS...

With NixOS, you can even reinstall your system on every boot: https://grahamc.com/blog/erase-your-darlings (I don't have this setup myself)

You can use Nix as a better docker build, see https://grahamc.com/blog/nix-and-layered-docker-images or https://nixery.dev/.