Isovalent Blog

Isovalent | Multiple roles | Mountain View (US), Zürich (CH), or Remote We're the company behind the open source Cilium project (https://cilium.io) (11K stars on GitHub) providing eBPF-based networking, observability, and security for container workloads and clusters. We have an amazing and in-demand product using revolutionary technology and are looking for top talent to help us build and explore all of its possibilities. We're remote-first, mainly in the EU and US timezones. If you're interested please apply through our careers site https://isovalent.com/careers and mention Hacker News in your application. Keywords for searchers: open source, Go/Golang, eBPF, C, C++, Kubernetes, networking, OpenShift, Linux kernel, performance, CI, SRE, technical writing, marketing, community advocate

For getting started with eBPF performance mindset, I normally recommend Brendan Gregg's book just to see what's possible: - https://www.brendangregg.com/bpf-performance-tools-book.html And as a related activity, you could just install the bcc-tools package (on RHEL clones) and check out the /usr/share/bcc/tools directory to see what's already implemented (on latest Ubuntu, these tools seem to be installed in /usr/sbin, etc, but you could "find /usr -name *bpfcc" to get a list of eBPF tools already installed there (and test/man some more interesting ones). For the bigger picture and other eBPF uses like networking, I'd get Liz Rice's eBPF book (free download): - https://isovalent.com/books/learning-ebpf/ But the most valuable resource for me when I took the leap from writing bpftrace one-liners to more sophisticated modern eBPF programs was (and still is) Andrii Nakryiko's blog with examples of modern BPF programming: - https://nakryiko.com/

The whole BPF verifier and development process is so botched, it's ridiculous. It's like maintainers decided to make this as hard as possible out of pettiness and "they have to use C APIs instead" or something. - Loading an eBPF module without the CAP_BPF (and in some cases without the CAP_NET_ADMIN which you need for XDP) capabilities will generate a "unknown/invalid memory access" error which is super useless as an error message. - In my personal opinion a bytecode format for both little endian (bpfel) and big endian (bpfeb) machines is kinda unnecessary. I mean, it's a virtual bytecode format for a reason, right!? - Compiling eBPF via clang to the bpf bytecode format without debug symbols will make every following error message down the line utterly useless. Took me a while to figure out what "unknown scalar" really means. If you forget that "-g" flag you're totally fucked. - Anything pointer related that eBPF verifier itself doesn't support will lead to "unknown scalar" errors which are actually out of bounds errors most of the time (e.g. have to use if pointer < size(packet) around it), which only happen in the verification process and can only be shown using the bpftool. If you miss them, good luck getting a better error message out of the kernel while loading the module. - The bpftool maintainer is kind of unfriendly, he's telling you to read a book about the bytecode format if your code doesn't compile and you're asking about examples on how to use pointers…

I still think of eBPF as not being Turing-complete. There is still a complexity limit in the verifier. Even if someone did implement Game of Life by having the program set a timer to run itself. https://isovalent.com/blog/post/ebpf-yes-its-turing-complete/

I think (not 100% sure) Cillium [0][1] kinda already does this. This loophole is good for packet processing/routing and even introducing XDP based ACL to bypass any ip/nf tables and get that almost wire speed benefit. I use Cilium with these features for custom made k8s clusters with Talos OS without any kube-proxy. [0]https://docs.cilium.io/en/stable/operations/performance/tuning.html [1]https://isovalent.com/blog/post/cilium-netkit-a-new-container-networki…

Also commonly used for high-performance networking and security use cases, for example https://isovalent.com/blog/post/cilium-netkit-a-new-container-networki… Basically anywhere you'd previously need to write a kernel module but now can have user space run arbitrary kernel code that's secure and won't crash the kernel. You can also now write custom schedulers in eBPF with sched_ext.

Cilium (a Kubernetes CNI) can use netkit instead of veth bridges since netkit was introduced in the kernel https://isovalent.com/blog/post/cilium-netkit-a-new-container-networki…

This article from isovalent introducing netkit walks through the benefits and tradeoffs https://isovalent.com/blog/post/cilium-netkit-a-new-container-networki…

It's actually answered (indirectly) on that page: "My report "What is eBPF?" and in-depth book "Learning eBPF" are both available for download [0] from Isovalent or with your subscription to O'Reilly's learning platform. You can buy "Learning eBPF" from any good bookstore (support your local bookshop by ordering it there!)" IIUC, you need to give contact info on that page to get the PDF. So this [1] might be a better starting point. [0] https://isovalent.com/ebpf/ [1] https://ebpf.io/

IIRC systemd uses eBPF for some of its sandboxing/security features, so probably some of your services would be more vulnerable to exploits. There are other tools that use eBPF to mitigate specific Linux kernel vulnerabilities, or to do Linux kernel observation/tracing. https://isovalent.com/blog/post/2022-05-16-tetragon/

and also a whitepaper / report from Isovalent on the same topic https://isovalent.com/blog/post/2022-04-oreilly-security can be downloaded for free (after filling the form), or if you have access to O'Reilly's Safari library, it is https://learning.oreilly.com/library/view/security-observability-with/… there.

there is also a post announcing it https://isovalent.com/blog/post/2022-05-16-tetragon

I'm not going to spam this forum with a marketing pitch so I'll just refer to https://www.isovalent.com/product and add that you can buy a Cilium Enterprise distribution with enterprise specific add-ons from us.