HAProxy Blog

> Hard disagree. Markdown is the way to go. As other child comments have stated, this is entirely subjective, and many would disagree with you (myself included). I live primarily in the terminal, and the ability to seamlessly look up syntax, flags, features, etc. without ever leaving the keyboard is wonderful. Additionally, I agree with milesrout that I do not want emoji, animations, etc. in my manuals (nor my terminal). I want searchable text. I also agree with lupusreal that having a single large page, at least as an option, is ideal. A perfect example of the two approaches is HAProxy. Here [0] is their commercial site. It's very flashy, modern web, etc. I don't necessarily hate it, but it's definitely designed to get the attention who may not necessarily know what they want, and can be sold to. In comparison, here [1] is their community site. It is extremely Web 1.0, and I love it. Everything you need, nothing you don't. Similarly, their docs [2] are [3] logically [4] split, and manage to be readable, navigable, yet information-dense. I don't need a tutorial with code snippets every few paragraphs in my docs. > Additionally, developers are very much used to markdown, and most other docs are written in markdown (maybe some RST, but probably mostly markdown) so this is not great for the person writing the docs either. Web devs, perhaps, though I imagine there's more variety than you think. Kernel devs likely do not have the same opinion as you (though I do not wish to…

HAProxy, specifically the docs site, make a fantastic job of making the vast documentation accessible and navigable. https://docs.haproxy.org/2.4/configuration.html https://www.haproxy.com/documentation/haproxy-configuration-manual/lat… I see now there is a Beta of a newer design of the docs at .com which, looking at it now, I hope they just scrap.

> It's amazing how people having visibly never dealt with high loads I've built multiple systems at 1M+ r/s and Tb+ scale. > The case where ports are quickly exhausted is with long connections, typically WebSocket Yes, HTTP2 is great for websockets. I was never advocating against it. The comment I was replying to was under the false assumption that you needed an outbound backend connection for every incoming connection. All of his concerns are solved problems in any modern open source load balancer. See https://www.haproxy.com/blog/http-keep-alive-pipelining-multiplexing-a… ;)

Yep - my understanding of https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations… is that Kubernetes doesn't usually control that load balancer (nor should it, since you could accidentally tell Kubernetes to take down the control plane LB, then not be able to get it back up again!). I suppose one could set up a "fast pass lane" kubeconfig that adds a header that haproxy would understand, and route to a priority class in its queue with e.g. https://www.haproxy.com/documentation/haproxy-configuration-tutorials/… . But there's no easy `kubectl --with-priority` (or, to my knowledge, good guidelines for the various gitops solutions) that follows this pattern out of the box.

HAProxy 2.8 has improved[0] Let's Encrypt integration by using acme.sh (so you can get rid of certbot). It still needs a cron job/systemd timers to do renewal of certs but acme.sh is just bash script so you don't need (extra?) system deps for installing it, while certbot requires Python and 12 python libs on my system (Fedora). And nowadays the "recommended" way to get certbot is via Snap (package manager)... [0] https://www.haproxy.com/blog/haproxy-and-let-s-encrypt

We'd love to hear about different use cases on our next HAProxyConf. both community and enterprise environments. We just wrapped up HAProxyConf 2022 in Paris (https://www.haproxy.com/blog/haproxyconf-2022-recap/) and are starting to plan for next one. The Call for papers hasn't been announced yet but feel free to shoot as an email at [email protected].

Kong since 0.x versions. We run into scale issues (database tuning parameters, caching, database replication) so I agree it would need a small team owning it. We actually use so little of the features that we consider building our own or look for other solutions. https://www.haproxy.com/user-spotlight-series/building-a-global-pop-ne… use haproxy as gateway. Possible because it's a single service, relatively simple authentication and users are fine without custom error responses. That might be worth looking into.

> Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". I wouldn't call it the best way; Rather a good way because Kubernetes does encapsulate the really good bits from scalability, development, security and reliability aspect. It's not a panacea but if you have team bandwidth to run k8s cluster, it's definitely worth a look. <3 HAProxy. It's a solid piece of software tested to the teeth over the years and is great; Here's the thing: You can run it as your preferred ingress controller too :) https://www.haproxy.com/blog/dissecting-the-haproxy-kubernetes-ingress…

I loved Chef after we got familiar with it. It was nice, cute, had solutions for a lot of problems. (foodcritic, kitchen, rubocop, etc.) But it simply is a solution in search of a problem in today's container orchestrator/platform world. HAproxy is amazing, but a k8s Ingress service [0] has a nice API, so I don't have to run Chef to add a new vhost. Yet I can persist the config in a YAML. (Or json, or whatever.) [0] which can of course be backed by HAproxy down below, but traefik has dynamic config; though haproxy2 will do that too https://www.haproxy.com/blog/haproxy-2-0-and-beyond/

It would be awesome if you guys could take the time to respond to the applications that you have actually received instead of ignoring applicants without feedback and asking for more CV's (which you need to process) on HN. You do seem to have a privacy policy[1] in place mentioning GDPR and hiring - it would be great if you could stick to the rules you claim to comply with on your site! thank you! [1] https://www.haproxy.com/privacy-policy/

In our maps blog post we have a section where we talk about applying different rate limits for different paths (https://www.haproxy.com/blog/introduction-to-haproxy-maps/#rate-limiti… For this I'd change the path fetch to a req.hdr(authorization) (or however you send API keys) to limit per API key. You could also add a 20/second with another stick table using the methods in this DDoS post (just with a string key to store API key rather than ip/src to store the IP address). You can put HAProxy directly on a box running another service to provide protection/reporting if you don't want to replace or add HAProxy to your ELB's (though sometimes that just adds too much complexity), we also talk about using HAProxy and ELB in our AWS blog post (https://www.haproxy.com/blog/haproxy-amazon-aws-best-practices-part-1/…

Will Tarreau of haproxy has written an insightful post on the same topic: "Truly Seamless Reloads with HAProxy – No More Hacks!" https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-mo…

You could start by taking a look at the Squid web site (http://www.squid-cache.org). Also the HAProxy web site (http://www.haproxy.org). There's some useful background info there. There's also a commercial entity behind HAProxy with some additional info (https://www.haproxy.com). A nice intro to HAProxy here: https://www.digitalocean.com/community/tutorials/an-introduction-to-ha… Also take a look at the Wikipedia pages for Squid and HAProxy. With regards books, the ones I looked at seemed a bit dated.

Why has HaProxy still such a 20 year old website? http://www.haproxy.org And then there is a blog on a new http://www.haproxy.com site. Is there a comparision of HaProxy, Varnish, Squid, Traffic Server (ATS), Nginx, lighttpd, etc for typical scenario.

AWS is certainly on the list of "Who is using HAProxy" here: https://www.haproxy.com/

A few notes on Power-Of-Two-Choices (Po2C) which is part of the solution described in the paper. NGINX appears to have Po2C available as an option on the "random" balancing algorithm. [1] It only considers load (requests in flight), unlike the paper which also considers recent latency (and does many other things). However NGINX "Plus" (the paid version) appears to have a couple additional settings that use actual traffic as latency probes (not asynchronous synthetic probes as described in the paper). [also 1] It's not fully clear to me but I think if you use these options it's going to only use latency, not load. The paper described in this post actually uses both load (RIF) and latency. Eager to try these out. Also, I noted that HAProxy author and kernel maintainer Willy Tarreau wrote an article in 2019 about adding Power-of-two-choices to HAProxy. [2] He builds a test bed with a few micro services and compares several balancing algorithms to Po2C, considering response time, throughput, and max load. One conclusion is that "At the very least, Power of Two is always better than Random. So, we decided to change the default number of draws for the Random algorithm from one to two, matching Power of Two." However, in the response time data, the test appears to consider only average response time. I would love to see the p90 and p99 tail latency of these tests. The main focus of the paper in this post is minimizing p90/99, not average latency. Tarreau also helpfully…

When reading along: Keep in mind that I'm not a core developer and thus are not directly involved in development, design decisions, or roadmap. I have some understanding of the internals and the associated challenges based on my contributions and discussions on the mailing list, but the following might not be entirely correct. > I wonder if a disk-writing process could be spun out before dropping privileges? I mean … it sure can and that appears the plan based on the last comment in that issue. However the “no disk access” policy is also useful for security. HAProxy can chroot itself to an empty directory to reduce the blast radius and that is done in the default configuration on at least Debian. > but I don't see why disk accesses need to block requests My understanding is that historically Linux disk IO was inherently blocking. A non-blocking interface (io_uring) only became available fairly recently: https://stackoverflow.com/a/57451551/782822. And even then it's a operating system specific interface. For the BSD's you need a different solution. If your process is blocked for even one millisecond when handling two million of requests per second (https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-r… then you drop 2k requests or increase latency. > or why they have to occur in the same thread as requests? “have” is a strong word, of course nothing “has” to be. One thing to keep in mind is that HAProxy…

Postman supports dynamic variables for tests [1]. Of course nothing requires you to use postman, you could just as easily script this with just ruby and curb/httparty. The reason for this is that a lot of api endpoints support a large number of parameters, and we want to be sure that we test a decent sample of real world usage instead of only a few examples that we can think of to implement in controller/request tests. We can take a month’s worth of request logs from production, deduplicate them, run the requests against prod and record responses, then run same requests against stage and verify responses match even after you rewrite half the backing queries. This helps a lot with peace of mind when we deploy :) For another, more advanced approach to this idea, see traffic mirroring/shadowing [2]. [1]: https://learning.postman.com/docs/writing-scripts/test-scripts/ [2]: https://www.haproxy.com/blog/haproxy-traffic-mirroring-for-real-world-…

Thanks for the feedback. I've provided a few replies below but I think you have made some great points and I will share them internally. > - the site does not do a good job of describing the products and what they do. The page does lead off by telling you it is a software load balancer, but I agree, we can do a better job of giving more information on the product itself. Most people who pursue HAProxy Enterprise are already familiar with HAProxy. With that said, we're actually working on a complete overhaul to this page and I believe the new version will address this. > - it is hard to find requirements from the first page. This is just a reference point and typically organizations are working closely with a sales engineer. > - is there a simpler HA setup than the 3 layer one here: A simpler version is found under active/standby here [1] [1] https://www.haproxy.com/documentation/hapee/2-2r1/high-availability/ac…

https://www.haproxy.com/blog/state-of-ssl-stacks According to this one should not be using v3 at all..

Since 2020, based on hostname, routing is possible to different ssh backends https://www.haproxy.com/blog/route-ssh-connections-with-haproxy from the blog > Did you know that you can proxy SSH connections through HAProxy and route based on hostname?

Regarding HAProxy, they ended up using AWS-LC in their new Debian/Ubuntu “performance” packages: https://www.haproxy.com/blog/fresh-from-aws-reinvent-supercharging-hap…

This article still relevant https://www.haproxy.com/blog/state-of-ssl-stacks

The whole 3.0 rewrite is a massive regression in all ways possible - they deprecated the old engines and replaced them with providers, and they are not that much easier to work with as a developer (I hope that providers are at least easier for the maintainers to handle) and the library is a lot more runtime dynamic (for some reason). This has resulted in mutex explosion and massive performance regressions in every facet. haproxy has an amusing article on the topic. https://www.haproxy.com/blog/state-of-ssl-stacks People who need cryptography but on the openssl API should be using aws-lc and seek a TLS stack elsewhere.

>What would some good examples of those be? There is haproxy-protection, which I believe is the basis of Kiwiflare. Clients making new connections have to solve a proof-of-work challenge that take about 3 seconds of compute time. Enterprise: https://www.haproxy.com/solutions/ddos-protection-and-rate-limiting FOSS: https://gitgud.io/fatchan/haproxy-protection

It's in beta, but HAProxy has a gateway product: https://www.haproxy.com/blog/announcing-haproxy-unified-gateway-beta

I use HAproxy to get HTTP/3. https://www.haproxy.org/ https://haproxy.debian.net/ https://www.haproxy.com/blog/how-to-enable-quic-load-balancing-on-hapr…

Willy from HAProxy has a good write-up on this. In their benchmarks, least-connections usually beat P2C, but P2C was never the worst and is arguably a saner default when least-connections isn’t available. The article link: https://www.haproxy.com/blog/power-of-two-load-balancing

Between this and https://www.haproxy.com/blog/state-of-ssl-stacks, I think we need to start accepting the idea that OpenSSL is not the right way forward for anything performance sensitive. Given how aws-lc powers both of these articles, I'm curious how Rustls compares to s2n-tls - AWS's TLS library to go along with aws-lc.

I wish they included details on how they ran these benchmarks, like they did last year [1]. I'd like to take a look and try to understand why there's such a big difference in handshake performance. I wouldn't expect single threaded handshake performance to vary so much between stacks... it should be mostly limited by crypto operations. Last time, they did say something about having a cpu optimization for handshaking that the other stack might not have, but this is on a different platform and they didn't mention that. I'd also be interested in seeing what it looks like with OpenSSL 1.1.1, given the recent article from HAProxy about difficulties with OpenSSL 3 [2] [1] https://www.memorysafety.org/blog/rustls-performance-outperforms/ [2] https://www.haproxy.com/blog/state-of-ssl-stacks