Use the HAProxy Runtime API to update SSL certificates in HAProxy without a reload.
When you route traffic through an HAProxy load balancer, you gain the ability to terminate SSL at the load balancer. HAProxy encrypts communication between the client and itself and then sends the decrypted messages to your backend servers, which means less CPU work on the servers because there’s no encryption work left to do. HAProxy adds extra SSL functionality too including SNI for choosing the right certificate, ALPN for negotiating the application protocol, OCSP stapling for prefetching certificate revocation statuses, and settings for disabling obsolete versions of SSL and TLS.
There had been just one shortcoming prior to version 2.1: When you renewed an SSL certificate, you had to reload HAProxy before it would pick up the change to the certificate file on disk. In itself, this isn’t so bad. HAProxy supports hitless reloads and guarantees that no traffic will be dropped during a reload. However, some users need to update certificates often—sometimes thousands at a time—and each time they would need to reload HAProxy in order for it to detect the changed files. When that happens, there could be a noticeable increase in memory usage because of all of those certificates being uploaded into memory in both the new process and the process that’s winding down.
Beginning with HAProxy version 2.1, that’s no longer the case. You can now use HAProxy’s Runtime API to update SSL certificates in HAProxy’s memory without needing to reload or restart the service. Version 2.1 added the ability to update existing certificates and version 2.2 made it even better by adding the ability to add and remove certificates that had not been previously loaded into memory. So, you can now add, replace, or remove certificates in the HAProxy process’s memory, which means you don’t need to reload HAProxy, which, in turn, means there’s less memory overhead when updating certificates.
Dynamic SSL Certificate Storage
One of the ways that HAProxy keeps its best-in-class performance and security is by reading from the filesystem at startup and then never again during its entire lifetime as a running process. That removes the possibility of slow filesystem I/O while HAProxy is doing the important work of proxying traffic and avoids a whole class of security vulnerabilities. However, it requires having to reload the process to pick up configuration changes or changes to other files. The HAProxy Runtime API gets around that by allowing you to perform certain actions dynamically. Now, SSL certificate management is one of those exceptions.
Suppose that you have an SSL certificate on the HAProxy server at /etc/haproxy/certs/site.pem. Assume that this file is referenced in the HAProxy configuration by having the following frontend configuration where the bind line points to the /etc/haproxy/certs directory. HAProxy will load all of the certificates found in that directory and choose the right one for a request based on SNI information:
Now, suppose that you’d like to replace this certificate with one that you have on your local workstation. Start by issuing the following API command to start a transaction that contains your new file. Note that I’m using socat to send these commands to the IP address and port where the remote HAProxy Runtime API is listening:
The set ssl cert command reads the contents of the new file and begins a transaction on the HAProxy server. By starting a transaction, you gain the ability to approve or cancel the operation before it’s final. After executing the command, you can check the status of the transaction with the show ssl cert command:
In the output of the show ssl cert command, you can see that the pending file has an asterisk before its name. You can get more information by adding the name of the file prefixed with an asterisk:
To finalize uploading the certificate into HAProxy’s memory, use the commit ssl cert command:
You can also use abort ssl cert if you decide to cancel the change instead.
After committing the transaction, run the show ssl cert command again with the name of the file, but this time without the asterisk, to see that the certificate has been updated. You should see that the Status field now says Used.
Although updating a certificate in memory means you don’t need to reload HAProxy, it’s a good idea to store the file on the HAProxy server so that when you do perform a restart or reload, HAProxy will pick up the new file at startup, rather than reverting back to a stale version that’s still on disk. There are a number of ways to do this, such as by using Rsync, SCP, or SFTP to transfer the files to the remote HAProxy server. For example, you could use this one-line SCP command to do it:
Then, when you do eventually restart HAProxy, the new file will be there to be loaded.
Bonus Feature: Updating CRT Lists
HAProxy 2.2 also includes a feature that lets you update CRT lists on the fly. If you aren’t familiar with them, CRT lists are text files that describe the SSL certificates that you’d like to use in your HAProxy configuration. You can store CRT list files in the same directory as your HAProxy configuration file. Here is an example CRT list, which I’ve stored at /etc/haproxy/crt-list.txt:
As shown here, each line can include attributes about each certificate, such as the preferred cipher suites, ALPN, minimum and maximum SSL versions, and a revocation list. See the documentation to learn all of the arguments that a line in a CRT list accepts. The last argument on the second line, test.local, is the SNI value to use for that certificate. If you have many certificates, then a CRT list will help cut down on the noise in your HAProxy configuration file and keep it simpler.
In your HAProxy configuration, you would reference the CRT list on a bind line with the crt-list argument:
You can now use the Runtime API to add or change lines in a CRT list file. First, if the certificate is new, use the new ssl cert command to create an empty slot for the certificate in HAProxy’s memory:
Next, begin a transaction to upload the certificate into that slot by using the set ssl cert command:
Then, use commit ssl cert to finalize the upload:
Now, to add a new line to the CRT list, use the add ssl crt-list command. Note the following syntax for setting the extra ALPN attribute and SNI value:
Finally, you can use show ssl crt-list to verify that the CRT list was updated correctly:
Conclusion
The new Dynamic SSL Certificate Storage that was introduced in version 2.1 and expanded in 2.2 lets you update SSL certificates dynamically without requiring a reload of the HAProxy service. It has a built-in transaction system so that you can verify your changes and then decide whether to commit or cancel them. For users who store thousands of certificates, this can make all the difference in memory usage, allowing them to run HAProxy on lighter spec’d machines.
Want to stay up to date on similar topics? Subscribe to this blog! You can also follow us on Twitter and join the conversation on Slack.
HAProxy Enterprise is the world’s fastest and most widely used software load balancer. It powers modern application delivery at any scale and in any environment, providing the utmost performance, observability, and security. Organizations harness its cutting edge features and enterprise suite of add-ons, backed by authoritative expert support and professional services. Ready to learn more? Contact us and sign up for a free trial.
