Cilium Blog

Isovalent | Multiple roles | Mountain View (US), Zürich (CH), or Remote We're the company behind the open source Cilium project (https://cilium.io) (11K stars on GitHub) providing eBPF-based networking, observability, and security for container workloads and clusters. We have an amazing and in-demand product using revolutionary technology and are looking for top talent to help us build and explore all of its possibilities. We're remote-first, mainly in the EU and US timezones. If you're interested please apply through our careers site https://isovalent.com/careers and mention Hacker News in your application. Keywords for searchers: open source, Go/Golang, eBPF, C, C++, Kubernetes, networking, OpenShift, Linux kernel, performance, CI, SRE, technical writing, marketing, community advocate

There are still people working on packet filtering! The BPF enhancements are benefiting lots of different areas of the Linux world at the same time. Cilium [1] is a great example of how the new capabilities of BPF can be used for more complex packet filtering, such as understanding higher level protocols. [1] https://cilium.io/

Talos Linux + Kubernetes, Rook (Ceph) storage, Cilium for networking, K8s cron and Tekton for cicd pipelines and misc tasks. There are Helm charts for all the software that would run on this. Can run on three servers, more servers can always be added. https://talos.dev https://rook.io https://cilium.io https://tekton.dev

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ (Networking) https://rook.io/ (Persistent Storage)

q3k has mentioned metallb+bgp, which is basically in-cluster implementation of LoadBalancer Service type (bgp speakers are running on k8s nodes and announce /32 routes to nodes based on configuration), but it doesn't provide an answer for "stabilizing" ecmp connections when there are changes to backends. There has to be something "behind" metallb[1] that will handle not only stable hashing for connections, but keep forwarding "in-flight" flows (like established tcp sessions) to correct backends, even if packets arrive on different ingress nodes. It seems cilium has some solution for that[2] (by both bundling metallb, and having maglev-based loadbalancer implementation) but I haven't had time to dig into it, so I was curious if someone else has solved it and would be willing to share stories from the front. This is one of those rough edges around kubernetes deployments in bare metal environments and I'd love to see what can be done to make it more robust. [1] metallb only really announces IPs so that "behind" is probably just CNI that actually handles traffic [2] https://cilium.io/blog/2020/11/10/cilium-19#maglev

Cilium's container networking and security product [0]; Facebook's Katran [1]; Netflix's flowsrus (not public yet, but see my tcplife tool in BCC[2]). That's just the beginning. It's not just for performance, it's also security, as while BPF programs run in kernel-mode they have a limited and secured API for interacting with the system (BPF helpers). Since extended BPF is in the Linux kernel (and will be in other kernels in the future), everyone is getting it, and we'll see more use cases over time. In some ways it's like the birth of JavaScript for the browser, and all the new applications it made possible. But it goes further than that: we could still analyze and debug JavaScript applications using traditional tools. But BPF programs are neither process-space or kernel routines, and are outside the view of everything. No visibility in ps(1), top(1), or lsmod(8). We're having to create new tools to even see what's running on the CPUs. Every performance monitoring product that shows a process table with CPU consumption will now need a BPF program table as well. [0] https://cilium.io/ [1] https://github.com/facebookincubator/katran [2] http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplife.html

You are right that cilium is a CNI but one of its many features is also providing a sidecar-less service mesh using eBPF. https://cilium.io/use-cases/service-mesh/

Cilium's Load Balancer became Maglev based last fall: https://cilium.io/blog/2020/11/10/cilium-19#maglev

I'm far more excited about dual stack support: https://cilium.io/blog/2021/05/20/cilium-110#dualstack

>Pod network traffic shaping Have you considered EDT-based rate limiting for Pods? This should scale well compared to TBF or HTB. Cilium developers have integrated this natively: https://cilium.io/blog/2020/11/10/cilium-19#bwmanager

No video yet, but here you have the paper. http://vger.kernel.org/lpc-networking2018.html#session-10 On the other hand, Cilium.io made a good summary of that. https://cilium.io/blog/2018/11/20/fb-bpf-firewall

hi, what do you think about the approaches of CiliumProject ? (https://opensource.googleblog.com/2016/11/cilium-networking-and-securi… They do have a Docker plugin (but does not work seamlessly with Swarm), but they claim to use BPF for superior performance https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacin… I think the Kubernetes and Swarm devs have generally leveraged IPVS as the next step from iptables. Any thoughts on going straight to BPF ?

How do you compare some of the features/goals of Cilium to istio's network policy? https://istio.io/blog/2017/0.1-using-network-policy.html Edit: Just came across this https://cilium.io/blog/istio/ :)

Cilium lives below userspace which makes it perform better than istio. This article has more information about the differences from the cilium developer's point of view. https://cilium.io/blog/istio/

For those interested in a really technical deep-dive on BPF, check out: http://cilium.readthedocs.io/en/latest/bpf/ If you have questions on BPF and Cilium for advanced firewalling, you can also ask them on the cilium slack: http://www.cilium.io/slack

One of the linked blog posts has a decent overview of the what & why of BPF: https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacin… The issue with microservices at scale vis-a-vis microservices is linear scaling of routing tables, very short-lived IP connections, and large update times for massive service maps. I think networking is complex in general, but the promise of this kind of integration (from my limited understanding), would be a clear reflection of services at the routing level instead of a hodgepodge of ports and IP addresses that provide little context or meaning. Having lost a few half-days learning iptable nuances in our Kubernetes cluster I can see the benefits in having an integrated stack with less impedance between architecture and routing. Generally we don't eliminate complexity, we just move it from one place to another. I imagine this is a case where a more complex service implementation could provide a simpler user experience.