HAProxy and Let’s Encrypt: Improved Support in acme.sh

TLS certificates are crucial for the secure flow of web traffic, but managing these digital certificates can pose a challenge. Using an ACME-based certificate authority like Let’s Encrypt can automate and simplify the management of issuing these certificates. 

Certificate management in HAProxy has steadily improved over the years, allowing it to become more flexible and load certificates without restarting. This improved certificate management has further been enhanced by HAProxy developer contribution to acme.sh to integrate smoothly with HAProxy.

Now, with the release of HAProxy 2.8, we’re sharing the progress we’ve made on this integration so far. In this blog post, we break down what TLS certificate management is all about and how you can start taking advantage of the latest support improvements with the Let’s Encrypt client, acme.sh.

What is TLS Certificate Management?

Certificate management is the monitoring and handling of digital certificates to ensure they are up to date and renewed once expired. It oversees that your web application’s identifying information is always legitimate and accessible for clients accessing your services.

Transport Layer Security (TLS) certificates are a type of digital certificate that play an integral role in facilitating secure communication between servers and clients. When a client initiates a connection, a handshake occurs with the server where the client verifies the validity and authenticity of the TLS certificate before establishing a secure, TLS-encrypted connection.

If a certificate happens to be expired, the client will identify your web application as a security risk—an issue no business wants to have. Certificate management helps avoid this by automating the timely renewal of TLS certificates, protecting your business from mistakes, and ensuring your web applications are always identified as a trusted service.

Managing certificates and their expiration can be challenging, especially when it comes to scale and automation. Without an automated certificate management process, the manual management of certificates can be difficult to keep track of, especially when managing multiple web applications.

The process of certificate management can be facilitated by the interaction between acme.sh, an ACME client, and Let’s Encrypt, a certificate authority. When issuance or renewal is required, acme.sh automatically oversees the management and deployment of certificates via Let’s Encrypt (albeit with some manual work to get started). The combination of the client, acme.sh, and the certificate authority can simplify certificate management for your web applications.

What is Let's Encrypt, and why should you use it?

Let’s Encrypt is a certificate authority that provides the digital certificates needed to enable HTTPS for application delivery.

Let’s Encrypt, a free, open-source certificate authority, automates the process of issuing TLS certificates. The core principle behind Let’s Encrypt is that the service is provided for the public’s benefit. Let’s Encrypt makes obtaining and managing certificates painless. It is completely free to use and is based on open standards, which, if you don’t know already, HAProxy Technologies is passionate about.

Most importantly, Let’s Encrypt offers three key benefits for your application delivery.

• Accessibility. Financial barriers are stripped away when it comes to using Let’s Encrypt, meaning you incur no additional expenses with the certificate authority service.

• Automation. The streamlining of the issuance and renewal of certificates simplifies the certification process and saves your team time.

• Community Support. Let’s Encrypt is backed by a large group of contributors, resources, and documentation, bringing invaluable support to HAProxy users.

Improved Support for HAProxy with Let’s Encrypt

With the release of HAProxy 2.8, the ACME client acme.sh is able to inform HAProxy deployments about newly issued certificates, and HAProxy is able to start using the new certificates immediately without restarting the process. This improvement means that when issuing and renewing TLS certificates, the HAProxy service can continue to run uninterrupted. HAProxy versions 2.2 and newer can also take advantage of this improved support.

Previously, you had to restart HAProxy to apply changes when deploying TLS certificates. Now, the improved support enhances service availability and efficiency in certificate management, and upholds security standards. The seamless integration of HAProxy with certificates results in a smooth, disruption-free experience for secure connections.

Installation and Configuration

As discussed, acme.sh integrates smoothly with HAProxy. With it, users are able to start an HAProxy configuration without a certificate, generate certificates with acme.sh, and populate HAProxy with them. 

Here’s how to get started by running acme.sh as non-root. This procedure was written for Ubuntu 22.04, and while these instructions are tailored for Let’s Encrypt, acme.sh supports other ACME-compatible certificate authorities, with ZeroSSL being the default.

HAProxy Package Installation

Before starting, ensure HAProxy is up-to-date by installing the latest HAProxy packages available. The following instructions are tailored for the latest version, HAProxy 2.8.

acme.sh installation

While acme.sh is not available as a package, installing acme.sh is easy. In this tutorial, we run acme.sh with its own user, granting it the necessary permissions within the HAProxy group. This setup ensures that acme.sh can push certificates in the appropriate location.

First, on the HAProxy server, create the acme user:

Add the acme user to the HAProxy group:

Create the directory where we will install the acme.sh program: 

Then install acme.sh:

Lastly, install the deploy script fork capable of updating certificates without restarting HAProxy and without requiring root access. This step is necessary because the merge request has not yet been merged into acme.sh. We recommend keeping an eye out for HAProxy deploy hook updates. 

Generate your ACME account

In this setup, acme.sh allows HAProxy to act as a proxy that responds to Let’s Encrypt challenges. With HAProxy typically handling HTTP traffic, it makes sense to have it also handle the challenges. To accomplish this, HAProxy will need to know the hash of the public key associated with your Let's Encrypt ACME account. Ordinarily, you don't see this detail because the Let's Encrypt client, in this case acme.sh, handles it for you and receives challenges directly. However, that approach requires extra complexity since we would need to relay the challenges to the acme.sh client. Considering we want HAProxy to receive the requests and send back responses itself, we need to get the hash and store it as a variable in the HAProxy configuration file.

To generate your ACME account, switch to the acme user so the ACME account info will be stored in this user's home directory. Replace "youremail@example.com" with the email address where you want to receive TLS certificate renewal notifications.

Once the account is registered, note down the thumbprint as it will be used to configure HAProxy.

Note that we have set the server where we'd like to register an account to be letsencrypt_test, which is the Let's Encrypt staging server. It's a good idea to use this value while you test your setup. Once you've verified that everything is working, register again using a server value of letsencrypt (this server name is mandatory for successful configuration).

Configure HAProxy to Respond to HTTP Challenges

When configuring HAProxy, it’s important to understand that acme.sh operates in a stateless mode as an ACME client, meaning it does not generate response to Let's Encrypt challenges dynamically, but rather relies on HAProxy to handle the HTTP challenge. Once HAProxy is configured for the task, you won’t need to make any further changes. In order to achieve this, HAProxy must return a specific value during the challenge.

Start by creating the directory for certificates:

Edit /etc/haproxy/haproxy.cfg to add the challenge response and the thumbprint. Don't forget the stats socket—this is needed to enable the HAProxy Runtime API, which acme.sh will call in order to load certificates into HAProxy’s runtime memory:

The strict-sni keyword will allow you to start HAProxy with the empty directory, and %[path,field(-1,/)] uses the random string Let’s Encrypt sent as part of the HTTP-01 challenge.

Restart the HAProxy service:

Generate a certificate

Now you can generate a certificate with acme.sh and HAProxy will respond to the challenge (this is visible in your HAProxy logs). 

Switch to the previously created acme user and run the following command. Replace "example.com" with your own domain, which must resolve to an accessible, public IP address. Here again we use the Let's Encrypt staging server.

Certificate deployment

Once the certificate is generated, you can deploy it to the /etc/haproxy/certs directory and enable it in HAProxy without needing to reload. This is possible because you configured the Runtime API by adding a stats socket directive to your HAProxy configuration. 

Now, call the acme.sh --deploy command, setting the values for DEPLOY_HAPROXY_STATS_SOCKET and DEPLOY_HAPROXY_PEM_PATH to match the appropriate path of the Runtime API socket and the directory where the certificates will be stored. Be careful not to add a trailing slash in DEPLOY_HAPROXY_PEM_PATH, as it won't be able to match the HAProxy configuration.

This command will deploy the certificate to /etc/haproxy/certs and inject the certificate over the HAProxy Runtime API.

Once complete, you can check if the certificate was added in HAProxy with socat:

And then directly with curl:

Be sure to update your firewall settings to open port 443. Once things appear to be working, switch to using the Let's Encrypt production server.

Certificate renewal

The certificates can be renewed and updated automatically using the --cron option. It is recommended to use the crontab of the acme user, which we created with the --system flag, or use a systemd timer.

Crontab

Using the crontab is easy since this is integrated in acme.sh:

Systemd Timer

The following is inspired by GitHub.

Create a new systemd service by defining the file /etc/systemd/system/acme_letsencrypt.service:

Test the service:

Create a time in /etc/systemd/system/acme_letsencrypt.timer:

Enable the timer:

Conclusion

By following these instructions, you can take advantage of the improved integration of HAProxy with acme.sh and simplify your certificate management with Let’s Encrypt. Eliminating the need to restart HAProxy during certificate generation results in a smoother experience in establishing secure connections between the client and server, and in the end, maintains the overall security of your application delivery.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.