Facebook Open Source Microsites

Yeah, I'd love to have something like BTF or CTF used widely for machine-readable type information. (https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf-enhanceme… gives some further information there.) The limitations regarding macros sound like the biggest issue to me (both code-like macros and just simple defined names for values via #define). I'd love to see solutions for that. What do you think that would look like?

Again: this is not a kernel module. eBPF probes are meant to be Compile Once, Run Everywhere, that's their whole point! https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portabili… If you expect software to be future-bug-proof, well, I guess you live in a far better world than I do. If you advertise your software to be compatible with RHEL, but a glibc bug gets in and causes your sw to crash for a couple of days, before RHEL realises the problem and fixes it, does that mean your software should instantly no longer be advertised as RHEL compatible? That'd make things a lot more confusing, if you ask me.

I mean think of low priority service as services that are not latency sensitive (background jobs). you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup) it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup. this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it. Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup. to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.ht…

That's once more not an answer to my question… Regarding spontaneously reboots: I never ever experienced this on Linux besides once having a hardware defect. (I would check the RAM in your case first; Manjaro is an Arch derivative so it incorporates the latest SW but it's not so terribly unstable that it would crash every two month I guess; not sure of course. I'm avoiding Arch for exactly the reason not being stable enough cause it's extremely fresh SW). Regarding the unresponsiveness on low memory: That's an known problem caused by a live-lock in the kernel. It has a few solutions by now. Just have a look at Facebooks oomd (or one of the solutions working in a comparable way): https://facebookmicrosites.github.io/oomd/docs/overview

Its been a bit of a complicated problem but memory pressure metrics are now a thing: https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.htm…

>Btrfs is probably never the right decision And you're posting anti-btrfs FUD. https://lwn.net/Articles/824855/ https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html Even if it's the only deployment in the world, it's more than "never". They provide strong technical reasons for using it.

Yes. If you ship any eBPF programs that utilize kprobes and/or read kernel structures that change between kernel versions, you have portability problems. See https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portabili… for a good overview of the problem that CO-RE intends to solve.

Facebook uses tupperware and Btrfs snapshots to quickly setup a container filesystem that's also isolated. https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html

Facebook runs on btrfs: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html

Facebook, for one, uses Btrfs for its containers: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html

Which we now have on Linux :D https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.htm…

I wonder if Facebook's opensourcing of BPF [1] 4 months ago had negative affects on F5's competitive advantaages. BPF allows to run user code in kernel modules, so things like packet rewriting and traffic shaping are possible. [2] [1]https://news.ycombinator.com/item?id=18337429 [2]https://facebookmicrosites.github.io/bpf/docs/bpf-docs#tools-built-on-…

I didn't quite get this post; half these things are stuff that facebook uses, not that they themselves have created and open-sourced (e.g. BTRFS initially from Oracle)? And yet they've set up they're own site for BTRFS and all (https://facebookmicrosites.github.io/btrfs/)

Depends on your usage patterns. On 8GB I gave it (ZRAM) between 512MB to 768MB RAM, and later on just using whatever values for the relevant sysctls AntiX was using(or their scripts created dynamically during setup/startup) at the time. It isn't just vm.swappiness, there are others, and they changed, or got removed, and replaced by others, over the years. And all of them sort of interwoven, influencing each other via what's happening in vm-subsystem. Can have bad feedback loops if you do it wrong, and be counterproductive. So you have to experiment, and measure with different settings, for what you're running. Where measure can be anything from htop/atop over https://github.com/cdown/psi-notify , to https://github.com/noiseonwires/memory-pressure and so on. To understand what these sysctls are about, how they are related to each other, and to ZRAM, I'd recommend reading, or at least skimming https://chrisdown.name/2018/01/02/in-defence-of-swap.html , https://chrisdown.name/2019/07/18/linux-memory-management-at-scale.htm… , https://linuxblog.io/linux-performance-almost-always-add-swap-space , https://linuxblog.io/linux-performance-almost-always-add-swap-part2-zr… , https://linuxblog.io/linux-performance-no-swap-space , https://linuxblog.io/running-out-of-ram-linux-add-zram , https://github.com/ValveSoftware/SteamOS/issues/899 , https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performa… , https://github.com/CachyOS/CachyOS-Settings/pull/19 ,…

We published this paper "TMO: Transparent Memory Offloading in Datacenters" last year which covers some Linux memory management mechanisms that may be quite useful for providing reasonable estimations to application memory usage. We observed that the real memory footprint for applications depends on many factors: file access pattern, disk IO speed (especially if swap is enabled), ssd vs hdd, application latency sensitivity, etc. Instead of coming up with some overly complicated heuristic, we use the Linux kernel provided memory.pressure [0] metric via cgroup v2. It measures the amount of time spent waiting for memory (page fault etc). Then by slowly reclaiming memory from the application until its memory pressure hits some target (say 0.1%), we can claim that the steady state usage is the actual memory footprint. This may not be useful for PC but could be very useful for data center to track memory regression, and also to harvest disk swap without concerning too much about the cliff effect when the host runs out of memory and suddenly kernel pushes everything to swap space. [0] https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.htm…

> If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on. This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer. Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly. An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory. On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages: * allows you to…

Most examples of BPF code are written in a mix of Python and C using BCC, the "BPF Compiler Collection", which essentially treats all of LLVM and clang as a library callable from Python code. I can't get my head around using it that way, and have found it pretty straightforward to just write C programs, compiled with clang `-target bpf`. Until very recently, writing anything interesting this way required you to declare all functions inline, compile into a single ELF .o, and, of course, avoid most loops. But most of the kinds of things you'd write in BPF tend not to be especially loopy (you can factor most algorithmic code out into userland, communicating with BPF using maps). A big issue for this kind of development is kernel compat; struct layouts can change from release to release, for instance. This isn't a problem for us at Fly, because we just run the same kernel everywhere, but it's a real problem if you're trying to ship a tool for other people's systems. But that's changing with CO-RE; recent kernels can export a simplified symbol table in a BPF-legible format called BTF, and the leader can perform relocations. Facebook has written a bunch of good stuff about this: https://facebookmicrosites.github.io/bpf/blog/2020/02/20/bcc-to-libbpf…