Dynamic SSL Certificate Storage in HAProxy

Use the HAProxy Runtime API to update SSL certificates in HAProxy without a reload.

When you route traffic through an HAProxy load balancer, you gain the ability to terminate SSL at the load balancer. HAProxy encrypts communication between the client and itself and then sends the decrypted messages to your backend servers, which means less CPU work on the servers because there’s no encryption work left to do. HAProxy adds extra SSL functionality too including SNI for choosing the right certificate, ALPN for negotiating the application protocol, OCSP stapling for prefetching certificate revocation statuses, and settings for disabling obsolete versions of SSL and TLS.

There had been just one shortcoming prior to version 2.1: When you renewed an SSL certificate, you had to reload HAProxy before it would pick up the change to the certificate file on disk. In itself, this isn’t so bad. HAProxy supports hitless reloads and guarantees that no traffic will be dropped during a reload. However, some users need to update certificates often—sometimes thousands at a time—and each time they would need to reload HAProxy in order for it to detect the changed files. When that happens, there could be a noticeable increase in memory usage because of all of those certificates being uploaded into memory in both the new process and the process that’s winding down.

Beginning with HAProxy version 2.1, that’s no longer the case. You can now use HAProxy’s Runtime API to update SSL certificates in HAProxy’s memory without needing to reload or restart the service. Version 2.1 added the ability to update existing certificates and version 2.2 made it even better by adding the ability to add and remove certificates that had not been previously loaded into memory. So, you can now add, replace, or remove certificates in the HAProxy process’s memory, which means you don’t need to reload HAProxy, which, in turn, means there’s less memory overhead when updating certificates.

Dynamic SSL Certificate Storage

One of the ways that HAProxy keeps its best-in-class performance and security is by reading from the filesystem at startup and then never again during its entire lifetime as a running process. That removes the possibility of slow filesystem I/O while HAProxy is doing the important work of proxying traffic and avoids a whole class of security vulnerabilities. However, it requires having to reload the process to pick up configuration changes or changes to other files. The HAProxy Runtime API gets around that by allowing you to perform certain actions dynamically. Now, SSL certificate management is one of those exceptions.

Suppose that you have an SSL certificate on the HAProxy server at /etc/haproxy/certs/site.pem. Assume that this file is referenced in the HAProxy configuration by having the following frontend configuration where the bind line points to the /etc/haproxy/certs directory. HAProxy will load all of the certificates found in that directory and choose the right one for a request based on SNI information:

global
# Enable the HAProxy Runtime API
stats socket :9999 level admin expose-fd listeners
frontend fe_sitea
mode http
# listen for HTTP traffic
bind :80
# listen for HTTPS traffic, load certificates from a directory
bind :443 ssl crt /etc/haproxy/certs/
# redirect HTTP to HTTPS
http-request redirect scheme https unless { ssl_fc }
# the pool of servers
default_backend servers
backend servers
server s1 172.25.0.11:8080 check
server s2 172.25.0.12:8080 check

Now, suppose that you’d like to replace this certificate with one that you have on your local workstation. Start by issuing the following API command to start a transaction that contains your new file. Note that I’m using socat to send these commands to the IP address and port where the remote HAProxy Runtime API is listening:

# Start a transaction to update the certificate
$ echo -e "set ssl cert /etc/haproxy/certs/site.pem <<\n$(cat ./new_certificate.pem)\n" | socat tcp-connect:172.25.0.10:9999 -

The set ssl cert command reads the contents of the new file and begins a transaction on the HAProxy server. By starting a transaction, you gain the ability to approve or cancel the operation before it’s final. After executing the command, you can check the status of the transaction with the show ssl cert command:

# Check the pending transaction (asterisk means pending)
$ echo "show ssl cert" | socat tcp-connect:172.25.0.10:9999 -
# transaction
*/etc/haproxy/certs/site.pem
# filename
/etc/haproxy/certs/site.pem

In the output of the show ssl cert command, you can see that the pending file has an asterisk before its name. You can get more information by adding the name of the file prefixed with an asterisk:

$ echo "show ssl cert */etc/haproxy/certs/site.pem" | socat tcp-connect:172.25.0.10:9999 -
Filename: */etc/haproxy/certs/site.pem
Status: Unused
Serial: 1F5202E02083861B302FFA09045721F07C865EFD
notBefore: Aug 12 17:05:34 2020 GMT
notAfter: Aug 12 17:05:34 2021 GMT
Subject Alternative Name:
Algorithm: RSA2048
SHA1 FingerPrint: C2958E4ABDF89447BF0BEDEF43A1A202213B7B4C
Subject: /C=US/ST=Ohio/L=Columbus/O=Company/CN=example.local

To finalize uploading the certificate into HAProxy’s memory, use the commit ssl cert command:

# Commit the transaction so HAProxy detects the change
$ echo "commit ssl cert /etc/haproxy/certs/site.pem" | socat tcp-connect:172.25.0.10:9999 -

You can also use abort ssl cert if you decide to cancel the change instead.

# Abort the transaction and cancel the change
$ echo "abort ssl cert /etc/haproxy/certs/site.pem" | socat tcp-connect:172.25.0.10:9999 -

After committing the transaction, run the show ssl cert command again with the name of the file, but this time without the asterisk, to see that the certificate has been updated. You should see that the Status field now says Used.

$ echo "show ssl cert /etc/haproxy/certs/site.pem" | socat tcp-connect:172.25.0.10:9999 -
Filename: /etc/haproxy/certs/site.pem
Status: Used
Serial: 1F5202E02083861B302FFA09045721F07C865EFD
notBefore: Aug 12 17:05:34 2020 GMT
notAfter: Aug 12 17:05:34 2021 GMT
Subject Alternative Name:
Algorithm: RSA2048
SHA1 FingerPrint: C2958E4ABDF89447BF0BEDEF43A1A202213B7B4C
Subject: /C=US/ST=Ohio/L=Columbus/O=Company/CN=example.local
Issuer: /C=US/ST=Ohio/L=Columbus/O=Company/CN=example.local

Although updating a certificate in memory means you don’t need to reload HAProxy, it’s a good idea to store the file on the HAProxy server so that when you do perform a restart or reload, HAProxy will pick up the new file at startup, rather than reverting back to a stale version that’s still on disk. There are a number of ways to do this, such as by using Rsync, SCP, or SFTP to transfer the files to the remote HAProxy server. For example, you could use this one-line SCP command to do it:

$ scp new_certificate.pem username@172.25.0.10:/etc/haproxy/certs/site.pem

Then, when you do eventually restart HAProxy, the new file will be there to be loaded.

Bonus Feature: Updating CRT Lists

HAProxy 2.2 also includes a feature that lets you update CRT lists on the fly. If you aren’t familiar with them, CRT lists are text files describing the SSL certificates you’d like to use in your HAProxy configuration. You can store CRT list files in the same directory as your HAProxy configuration file. Here is an example CRT list, which I’ve stored at /etc/haproxy/crt-list.txt:

/etc/haproxy/certs/default.pem
/etc/haproxy/certs/test.local.pem [alpn h2 ssl-min-ver TLSv1.2] test.local

As shown here, each line can include attributes about each certificate, such as the preferred cipher suites, ALPN, minimum and maximum SSL versions, and a revocation list. See the documentation to learn all of the arguments that a line in a CRT list accepts. The last argument on the second line, test.local, is the SNI value to use for that certificate. If you have many certificates, then a CRT list will help cut down on the noise in your HAProxy configuration file and keep it simpler.

In your HAProxy configuration, you would reference the CRT list on a bind line with the crt-list argument:

frontend fe_main
mode http
bind :80
bind :443 ssl crt-list /etc/haproxy/crt-list.txt
http-request redirect scheme https unless { ssl_fc }
default_backend servers

You can now use the Runtime API to add or change lines in a CRT list file. First, if the certificate is new, use the new ssl cert command to create an empty slot for the certificate in HAProxy’s memory:

$ echo -e "new ssl cert /etc/haproxy/certs/new_certificate.pem" | socat tcp-connect:127.0.0.1:9999 -
New empty certificate store '/etc/haproxy/certs/new_certificate.pem'!

Next, begin a transaction to upload the certificate into that slot by using the set ssl cert command:

$ echo -e "set ssl cert /etc/haproxy/certs/new_certificate.pem <<\n$(cat ./new_certificate.pem)\n" | socat tcp-connect:127.0.0.1:9999 -
Transaction created for certificate /etc/haproxy/certs/new_certificate.pem!

Then, use commit ssl cert to finalize the upload:

$ echo -e "commit ssl cert /etc/haproxy/certs/new_certificate.pem" | socat tcp-connect:127.0.0.1:9999 -
Committing /etc/haproxy/certs/new_certificate.pem
Success!

Now, to add a new line to the CRT list, use the add ssl crt-list command. Note the following syntax for setting the extra ALPN attribute and SNI value:

$ echo -e "add ssl crt-list /etc/haproxy/cert-list.txt <<\n/etc/haproxy/certs/new_certificate.pem [alpn h2] mysite.local\n" | socat tcp-connect:127.0.0.1:9999 -
Inserting certificate '/etc/haproxy/certs/new_certificate.pem' in crt-list '/etc/haproxy/cert-list.txt'.
Success!

Finally, you can use show ssl crt-list to verify that the CRT list was updated correctly:

$ echo "show ssl crt-list /etc/haproxy/cert-list.txt" | socat tcp-connect:127.0.0.1:9999 -# /etc/haproxy/cert-list.txt
/etc/haproxy/certs/site.pem
/etc/haproxy/certs/test.local.pem [alpn h2 ssl-min-ver TLSv1.2] test.local
/etc/haproxy/certs/new_certificate.pem [alpn h2] mysite.local

Conclusion

The new Dynamic SSL Certificate Storage introduced in version 2.1 and expanded in 2.2 lets you update SSL certificates dynamically without requiring a reload of the HAProxy service. It has a built-in transaction system so that you can verify your changes and then decide whether to commit or cancel them. For users who store thousands of certificates, this can make all the difference in memory usage, allowing them to run HAProxy on lighter spec’d machines.

Want to stay up to date on similar topics? Subscribe to this blog.

HAProxy Enterprise is the world’s fastest and most widely used software load balancer. It powers modern application delivery at any scale and environment, providing the utmost performance, observability, and security. Organizations harness its cutting-edge features and enterprise suite of add-ons backed by authoritative expert support and professional services. Ready to learn more? Sign up for an HAProxy Enterprise free trial.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.