Navigating the complexities of Transport Layer Security (TLS) has long been a challenge for developers, with the intricacies of libraries like OpenSSL often proving difficult to master. For those seeking to secure their network traffic without compromising performance, the journey to robust TLS implementation can be daunting. This session delves into the evolution of TLS within HAProxy, as presented by William Lallemand, an HAProxy software engineer and maintainer of its TLS stack, offering a comprehensive look at how HAProxy has tackled these challenges.
This session explores the historical progression of TLS integration, from the early days of relying on external solutions like Stunnel and Stud, to the development of HAProxy's own native, high-performance TLS stack. You will gain insights into key advancements such as the Proxy Protocol and shared session caching, and explore modern features designed to streamline certificate management, including internal certificate caching, crt-lists, and dynamic operations using the runtime API.
Furthermore, the session highlights innovations like the crt-store and the new ssl-f-use keyword, demonstrating how HAProxy 3.2 offers unparalleled flexibility and elegance in configuring TLS options. It will also cover the integration of modern protocols like QUIC and the experimental ACME support, alongside a critical look at supported TLS libraries and their performance implications. By the end of this read, you'll have a clearer understanding of HAProxy's commitment to providing a complete, modern, and highly performant TLS stack, ensuring secure and efficient network communication.
Slide Deck
Here you can view the slides used in this presentation if you’d like a quick overview of what was shown during the talk.
I'm William Lallemand. I'm an HAProxy software engineer, and I'm one of the maintainers of the HAProxy TLS stack. I'm going to talk about TLS on HAProxy 3.2.
Here’s the history of TLS within HAProxy.
For HAProxy 1.4 and before:
Before the “TLS Everywhere” era
No native TLS support
Complex libraries
Event-driven model
“We will never implement TLS natively.” – Willy Tarreau
Stunnel
Forks, threads,
ucontextmodel
TLS was not mandatory in 2000. There was no native TLS in HAProxy.
As developers, we did not know much about TLS at this time. Libraries were complex. It was difficult to understand OpenSSL. It's still difficult. Most TLS implementations were using fork/thread models. HAProxy is an event-driven model, so we didn't know how to use SSL in HAProxy. We thought that the cryptographic operation would be too expensive for the non-blocking event-driven model. We thought that we would never implement TLS natively.
However, there were still users requiring TLS. These users were using Stunnel, which is a well-known tunneling program. It's a TLS frontend that you put in front of your TCP service. It basically encrypts everything, and it was available in fork, thread, and ucontext models. At the time, we were using the ucontext model, which is a kind of coroutine, but the architecture was really different from HAProxy.
The typical deployment looked like this:
You would have one Stunnel on the same machine as HAProxy, or you could have multiple Stunnels to support scaling.
Stunnel was difficult to use because HAProxy couldn't see the client IP. In the log, you saw only the IP of Stunnel, which was localhost, which was not meaningful or useful.
To address these issues, we had to improve Stunnel:
Proxy protocol
TLS stateful resumption
Scalability was limited
We created the Proxy protocol, a small header at the beginning of the connection that contains the client IP and port. There was also the problem of TLS resumption. At this time, it was mostly TLS 1.0 and TLS 1.1, and TLS resumption was stateful. For stateful resumption, you need a cache, and if you have multiple Stunnel processes, they need to share this cache. So we implemented a shared cache.
We didn’t resolve the scalability issue, but at that time, these Stunnel changes were sufficient.
A few years later the “TLS Everywhere” era began. Maybe you remember the green address bar in Firefox and Chrome? A lot more people wanted to have TLS on their servers, but we needed something more performant than Stunnel.
Stunnel replacement:
TLS everywhere
Event-driven
No fork/thread
Added Proxy protocol
Added shared-session cache
Later became Hitch (https://hitch-tls.org)
There weren't many alternatives at the time, but the Stud project appeared. Stud is an event-driven project. It's kind of like HAProxy. It didn't use fork and thread, so we started to work with it. We contributed the same features that we added in Stunnel: the Proxy protocol and the shared session cache. The project exists today as Hitch, which is maintained by Varnish Software.
So the deployment was pretty much the same, but you would be able to use only one process because it was more performant and more scalable.
We learned a lot from Stud.
Unfortunately, Stud was functionally limited. It was still a separate process in front of HAProxy, which meant it cost a lot of memory and CPU. The benefit, however, was that Stud served as a useful proof of concept, and working with Stud taught us how to implement the same functionality natively within HAProxy.
We started developing our own TLS stack in 2010, and it took four years to complete. The result is a fairly complete, high-quality stack. With our new TLS stack, deployments became much simpler. You needed only one HAProxy process and no other processes.
Once we had our complete TLS stack, we needed to improve the configuration and simplify management. Management issues included:
Users with hundreds of thousands of certificates
Internal certificate cache
Dynamic refresh
Dynamic
crt-list
We had some users with hundreds of thousands of certificates. It could be difficult for those users because HAProxy could take a lot of time to load, perhaps five or ten minutes. Similarly, a simple reload takes just as long. This could be a critical penalty when all you want to do is add or remove a single certificate.
We implemented a few features to make it easier.
We first implemented the internal certificate cache, which loads the certificate in memory only once. In this example configuration, the same certificate is repeated on each bind line. Without a cache, you have to load the certificate from disk each time. With a cache, however, the certificate needs to be loaded only once, resulting in a considerable reduction in loading time.
We also implemented the crt-list.
Implemented in HAProxy 1.5, crt-list allows you to list multiple certificates in a file along with associated information such as TLS options and ciphers. You can also specify filters that allow you to use a domain other than the one that is inside your certificate. For example, if your customer provides your certificate, you probably want to check if it's the right domain. You can specify the correct domain for the customer at the end of the line.
While the crt-list simplifies the task of specifying certificates, changes still require that you reload HAProxy. To enhance the process for updating certificates, we implemented the certificate operations in the HAProxy Runtime API.
In this example, we are creating a new certificate, called site1. We are reading it from the disk and committing it, so this is basically a small transaction. Then we add it to an existing crt-list.
The HAProxy Runtime API commands made the operations more dynamic, but it was still difficult to organize certificates. Previous versions of HAProxy needed the key and the certificate to be in the same PEM file. It was not convenient to separate the permissions of the file or to just put the certificates and the keys in different directories.
We solved these problems by implementing the crt-store.
In this example, we have site1.crt with its key configured, and we can specify options like the OCSP updates and other storage options. You can also use aliases. Here, the S2 alias allows you to use S2 in the configuration instead of the full name of the certificate. You can split the certificates per feature or per customer. For example, you have the web1 namespace there, but you could create a unique namespace per customer.
The crt-store is convenient for configuring storage. However, the only way to configure TLS options per certificate was by using the crt-list. Unfortunately, this solution is not convenient when dealing with hundreds of frontends because you need to create one crt-list file per frontend.
To solve this problem, we introduced the ssl-f-use keyword in HAProxy 3.2.
Using the ssl-f-use directive, you can specify a crt-list inline in your frontend configuration. You don't have any bind line limitations, so you can just put all your TLS options on the ssl-f-use line. It's also more dynamic because you can use all the crt-list operations on the stats socket.
This example shows that ssl-f-use provides an elegant way to deduplicate your configuration. For example, if you want to use TCP for HTTP/1 and /2, and QUIC for HTTP/3, you just have to put all your certificates on each ssl-f-use line, and they will apply on every bind line in the frontend. Without ssl-f-use, you would have to introduce duplicate entries for each certificate.
These enhancements provide a complete and modern TLS stack, providing unparalleled performance and security.
A recent addition has been the QUIC protocol on the frontend side. It's implemented with every library that we support. With OpenSSL 3.0, this task was difficult because there wasn't a QUIC API. With OpenSSL 3.5, however, there is full support for QUIC, including 0-RTT (Zero Round Trip Time Resumption).
With HAProxy 3.2, we implemented the ACME (Automatic Certificate Management Environment) protocol. It's an experimental feature for now. See the presentation Zero Downtime: Automating HAProxy Certificate Management with ACME.
With these and other changes, HAProxy has kept up with evolving stack technologies. We recommend that you update what you are using. For example, stop using TLS 1.0. Stop using SSL 3.0. A lot of algorithms and ciphers are evolving. There’s talk now of deprecating the standard Diffie-Hellman in favor of the elliptic curve Diffie-Hellman. Increasingly, users are choosing the elliptic curve DSA instead of RSA certificates. We also recommend that you stop using stateful resumption because of the performance cost. Almost universally, modern stacks support stateless resumption, so there’s no longer a need for stateful resumption.
Regarding libraries, we recommend that you upgrade any OpenSSL 3.0 version and use at least version 3.1, or use release 3.5.1 if available.
We support QuicTLS. It was just a patch set on top of OpenSSL, but it's evolving now. We support WolfSSL as well, which is an efficient library, but it uses a compatibility layer, so we don't have all of its features.
We support AWS-LC, which is, for me, the best library out there for HAProxy.
Here are some benchmark results:
Ranked by performance, the libraries fall in this order:
AWS-LC (highest TLS connections-per-second)
WolfSSL
OpenSSL 1.1
OpenSSL 3.4/3.5
OpenSSL 3.0
These figures are taken from our article, The State of SSL Stacks.
Looking ahead, the future offers a number of SSL technologies that we can explore, such as:
AWS-LC: We’ve selected AWS-LC as the best choice for security and performance, so we’ll be using it as we develop new packages for the community and for HAProxy Enterprise.
Rustls: We’ve been experimenting with rustls. We found the OpenSSL compatibility layer problematic, so we’ll be working with the rustls developers to implement improvements.
QuicTLS: It’s outgrowing its status as a mere patch set and becoming a complete, new library. There were some OpenSSL issues, but the QuicTLS team will be addressing them in the future.
ACME: We want to complete our ACME support. Currently, we can handle HTTP-01 and DNS-01 challenges with an external component, but we want to do more.
ECH: Encrypted ClientHello (ECH) enhances ClientHello by encrypting the domain that you're asking for.
kTLS: We also want to make progress on kTLS, which is the kernel implementation of TLS. This technology is particularly interesting if you want to use cryptographic acceleration from the NIC.
With these and other improvements, HAProxy can only improve its market-leading record for security and performance. Thank you for your attention.
