Enterprise modules

Update a certificate store from a remote source

On this page

This page applies to:

  • HAProxy Enterprise 3.4r1 and newer
  • HAProxy ALOHA 18.5 and newer

The Certificate Store Update module fills a named crt-store with certificates downloaded from one or more URLs, and fetches them again on an interval. A certificate store normally holds the certificates that were loaded from disk when the load balancer started. With this module, the store can also track certificates that live somewhere else.

The module supports fetching certificates from two types of URLs:

  • Fetch certificates from a JSON Web Key Set (JWKS) URL.
  • Fetch certificates from a URL that hosts PEM files, when the identity provider doesn’t host a JWKS URL.

Fetching certificates from a JWKS URL is appropriate when you need to verify that JSON Web Tokens (JWTs) have been digitally signed by the identity provider that issued them. A JWKS is a JSON document that contains one or more public keys, indexed with unique IDs, that match up with digital signatures. Providing multiple keys in a single document makes it possible to rotate signing keys without disrupting clients still using JWTs signed by an older key. That’s because each JWT embeds the ID of the key it was signed with, so the right key can be looked up in the JWKS document.

Ensure the keys set the x5c field

To be compatible with HAProxy Enterprise’s Certificate Store Update module, a JWKS should include an x5c field for each key, which contains the public key in X.509 certificate form.

The module can also fetch certificates from a URL that returns a single certificate PEM file, in case the identity provider doesn’t host a JWKS URL but lets you download the certificates directly. You wouldn’t have the same ability to safely rotate keys, since you wouldn’t get a document with multiple certificates.

Use this module when verifying the digital signatures of JWTs during an OAuth flow. You don’t need this module when configuring the OpenID Connect module, since that has a built-in feature for fetching keys from a JWKS URL.

Install the Certificate Store Update module Jump to heading

  1. Install the Certificate Store Update module according to your platform:

    Install the Certificate Store Update module
    nix
    sudo apt-get install hapee-3.4r1-lb-crtstore-update
    nix
    sudo apt-get install hapee-3.4r1-lb-crtstore-update
    nix
    sudo yum install hapee-3.4r1-lb-crtstore-update
    nix
    sudo yum install hapee-3.4r1-lb-crtstore-update
    nix
    sudo zypper install hapee-3.4r1-lb-crtstore-update
    nix
    sudo zypper install hapee-3.4r1-lb-crtstore-update
    nix
    sudo pkg install hapee-3.4r1-lb-crtstore-update
    nix
    sudo pkg install hapee-3.4r1-lb-crtstore-update

  2. In the global section of your configuration file, add a module-load directive:

    haproxy
    global
      ...
      module-load hapee-lb-crtstore-update.so
    haproxy
    global
      ...
      module-load hapee-lb-crtstore-update.so

    The default HAProxy Enterprise configuration already sets module-path to the module directory, so the module-load line is all you need.

    Try the fetch flow with the sample certificates

    The module package includes example JWKS documents, so you can exercise the fetch flow without a real identity provider. To test this:

    1. Serve the example JWKS documents from a local web server:

      nix
      cd /etc/hapee-3.4/crtstore_examples
      python3 -m http.server 8000
      nix
      cd /etc/hapee-3.4/crtstore_examples
      python3 -m http.server 8000

    2. Put the test configuration in its own file, so your main configuration is untouched:

      crtstore-test.cfg
      haproxy
      global
        log stderr format raw local0 info
        stats socket /tmp/crtstore-test.sock mode 600 level admin
        module-path /opt/hapee-3.4/modules
        module-load hapee-lb-crtstore-update.so
      crt-store sample_store
        update url http://127.0.0.1:8000/keys.jwk delay 5 retry-delay 5 format jwk
      crtstore-test.cfg
      haproxy
      global
        log stderr format raw local0 info
        stats socket /tmp/crtstore-test.sock mode 600 level admin
        module-path /opt/hapee-3.4/modules
        module-load hapee-lb-crtstore-update.so
      crt-store sample_store
        update url http://127.0.0.1:8000/keys.jwk delay 5 retry-delay 5 format jwk

    3. Start the load balancer. The log stderr line makes each fetch visible:

      nix
      sudo /opt/hapee-3.4/sbin/hapee-lb \
        -f /etc/hapee-3.4/crtstore-test.cfg
      nix
      sudo /opt/hapee-3.4/sbin/hapee-lb \
        -f /etc/hapee-3.4/crtstore-test.cfg

    4. After a few seconds, check the store in another terminal:

      nix
      echo "show ssl jwt" | sudo socat stdio /tmp/crtstore-test.sock
      nix
      echo "show ssl jwt" | sudo socat stdio /tmp/crtstore-test.sock

      The store lists one entry per key in the document:

      output
      text
      # filename
      @sample_store/<kid>
      output
      text
      # filename
      @sample_store/<kid>

    Three sample documents are installed:

    • keys.jwk and keycloak_keys.jwk are working JWKS documents from two providers.
    • invalid.jwk is deliberately not valid JSON. Use it to see the failure path: the fetch returns 200, nothing is inserted, and the log shows only an ordinary request line.

    A blank line instead of a list means nothing has been fetched yet. Check the log stderr output for the fetch, and run the command again after a few seconds.

Fetch certificates from a JWKS URL Jump to heading

This example demonstrates a portion of an OAuth configuration that verifies bearer tokens that clients present to the load balancer. The identity provider publishes its signing certificates as a JSON Web Key Set (JWKS).

  1. Declare a named certificate store and one update line per remote source:

    hapee-lb.cfg
    haproxy
    crt-store idp_store
      update url https://login.example.com/common/discovery/v2.0/keys delay 3600 retry-delay 60 format jwk
    hapee-lb.cfg
    haproxy
    crt-store idp_store
      update url https://login.example.com/common/discovery/v2.0/keys delay 3600 retry-delay 60 format jwk

    In this example:

    • url points to the provider’s key discovery endpoint. It is required and must come first.
    • delay 3600 refetches the URL every hour after a successful fetch.
    • retry-delay 60 retries a failed fetch after one minute.
    • format jwk reads the response as a JWKS document and names each certificate by its kid.

  2. Address the fetched certificates in your configuration. Each entry of the JWKS is stored under @<store_name>/<kid>, where kid is the key identifier from the JWKS document. Read the kid from the token’s header to build the name:

    hapee-lb.cfg
    haproxy
    frontend api
      bind :443 ssl crt /etc/hapee-3.4/site.pem
      mode http
      http-request set-var(txn.jwt) http_auth_bearer
      http-request set-var(txn.alg) var(txn.jwt),jwt_header_query('$.alg')
      http-request set-var(txn.kid) var(txn.jwt),jwt_header_query('$.kid')
      http-request set-var(txn.crt) str(),concat('@idp_store/',txn.kid)
      http-request deny content-type 'text/html' string 'Invalid token' unless { var(txn.jwt),jwt_verify_cert(txn.alg,txn.crt) -m int 1 }
    hapee-lb.cfg
    haproxy
    frontend api
      bind :443 ssl crt /etc/hapee-3.4/site.pem
      mode http
      http-request set-var(txn.jwt) http_auth_bearer
      http-request set-var(txn.alg) var(txn.jwt),jwt_header_query('$.alg')
      http-request set-var(txn.kid) var(txn.jwt),jwt_header_query('$.kid')
      http-request set-var(txn.crt) str(),concat('@idp_store/',txn.kid)
      http-request deny content-type 'text/html' string 'Invalid token' unless { var(txn.jwt),jwt_verify_cert(txn.alg,txn.crt) -m int 1 }

    In this example:

    • http_auth_bearer reads the token from the Authorization header and stores it in the txn.jwt variable.
    • jwt_header_query('$.alg') extracts the signing algorithm from the token header and stores it in the txn.alg variable.
    • jwt_header_query('$.kid') extracts the key identifier from the token header and stores it in the txn.kid variable.
    • str(),concat('@idp_store/',txn.kid) builds the certificate name in the store from the store name and the token’s own kid, so a token signed with a newly rotated key resolves to the certificate that the store fetched for it.
    • jwt_verify_cert(txn.alg,txn.crt) verifies the token signature against that certificate, and the http-request deny line rejects the request unless the converter returns 1.

  3. Confirm that the fetched certificates are in the store. The module marks them for token validation, so they appear in the list that show ssl jwt returns:

    nix
    echo "show ssl jwt" | \
      sudo socat stdio unix-connect:/var/run/hapee-3.4/hapee-lb.sock
    nix
    echo "show ssl jwt" | \
      sudo socat stdio unix-connect:/var/run/hapee-3.4/hapee-lb.sock
    output
    text
    # filename
    @idp_store/<kid>
    output
    text
    # filename
    @idp_store/<kid>

    Tip

    A certificate that’s missing from this list hasn’t been fetched yet. Wait a few seconds and run the command again, or check the url argument.

Fetch a PEM certificate Jump to heading

The pem format accommodates certificate sources that publish a single X.509 certificate directly, rather than exposing a JWKS document. An internal or legacy token issuer, for example, can publish its current signing certificate at a stable URL. The load balancer refreshes that certificate after each key rotation, without external distribution automation, and without a JWKS endpoint that exists only to distribute one public certificate.

Each argument follows from what the source publishes. The remote resource is already a certificate, so format pem inserts it as is. The document has no kid, so you must pass name, and setting name with format jwk is rejected. There is no key set to enumerate, so one URL supplies exactly one certificate.

hapee-lb.cfg
haproxy
crt-store tls_certs
  update url https://pki.example.com/edge.pem delay 86400 retry-delay 600 format pem name edge.pem
hapee-lb.cfg
haproxy
crt-store tls_certs
  update url https://pki.example.com/edge.pem delay 86400 retry-delay 600 format pem name edge.pem

Info

Setting name with format jwk isn’t valid and the load balancer will reject it.

In this example, the certificate is available as @tls_certs/edge.pem, and show ssl jwt lists it under that name, alongside any certificates that came from a JWKS document. Reference that name where the load balancer looks up a certificate at runtime, such as in the jwt_verify_cert converter:

hapee-lb.cfg
haproxy
frontend api
  bind :443 ssl crt /etc/hapee-3.4/site.pem
  mode http
  http-request set-var(txn.jwt) http_auth_bearer
  http-request set-var(txn.alg) var(txn.jwt),jwt_header_query('$.alg')
  http-request deny content-type 'text/html' string 'Invalid token' unless { var(txn.jwt),jwt_verify_cert(txn.alg,'@tls_certs/edge.pem') -m int 1 }
hapee-lb.cfg
haproxy
frontend api
  bind :443 ssl crt /etc/hapee-3.4/site.pem
  mode http
  http-request set-var(txn.jwt) http_auth_bearer
  http-request set-var(txn.alg) var(txn.jwt),jwt_header_query('$.alg')
  http-request deny content-type 'text/html' string 'Invalid token' unless { var(txn.jwt),jwt_verify_cert(txn.alg,'@tls_certs/edge.pem') -m int 1 }

Caution

You can’t use a fetched certificate as the server certificate on a bind line. A certificate fetched this way carries no private key, and a bind line resolves its certificates when the configuration is parsed, before the first fetch.

Reference Jump to heading

The update directive configures the remote sources that fill a certificate store. The converters below read certificates from that store, and the sections that follow describe how refreshes work and what the module cannot do.

update Jump to heading

The update directive fills and refreshes the certificate store that contains this line. Declare the line inside a named crt-store section, one line per remote source. A store can hold several update lines, each with its own URL and its own delay and retry-delay.

text
update url <url>
  [delay <seconds>]
  [retry-delay <seconds>]
  [format <jwk|pem>]
  [name <certname>]
text
update url <url>
  [delay <seconds>]
  [retry-delay <seconds>]
  [format <jwk|pem>]
  [name <certname>]
Argument Description
url <url> URL that the load balancer requests with a GET. It’s required, and must appear first. The load balancer resolves the hostname with its internal HTTP client.
delay <seconds> Delay in seconds between two successful fetches of this URL. The default is 86400, or 24 hours.
retry-delay <seconds> Delay in seconds between a failed request and the next attempt against the same URL. The default is 600, or 10 minutes.
format <jwk|pem> Format of the data returned by the remote source. The default is jwk.
name <certname> Name of the certificate. Mandatory when format is pem, and rejected when format is jwk.

Configuration errors

The load balancer rejects the configuration when:

  • a crt-store section has no name
  • an update line has no url argument or a second url argument
  • format is neither jwk nor pem
  • name is present with format jwk or missing with format pem

jwt_verify_cert Jump to heading

The jwt_verify_cert converter verifies the signature of a signed token against a certificate. Pass the algorithm from the token header and the certificate name, such as @idp_store/<kid>. See jwt_verify_cert in the configuration manual.

How certificate refreshes work Jump to heading

  • Soon after the load balancer starts, it requests each URL for the first time. If the remote source is unreachable at that moment, the store stays empty until a fetch succeeds, and the load balancer retries every retry-delay seconds.

  • The load balancer fetches each URL every delay seconds after a successful fetch, and every retry-delay seconds after a failed one. A response with a status other than 200 counts as a failure.

  • When a fetch or a parse fails, the certificates already in the store stay in place, and the load balancer retries later.

  • Only one URL is fetched at a time. Entries are ordered by their next update time, so a URL that fails doesn’t hold up the others.

  • A successful fetch replaces the certificate in the store. No reload is needed, and existing uses of the certificate pick up the new one.

  • delay and retry-delay are integers in seconds. Unit suffixes aren’t supported, so 1h is read as 1 second and a value starting with a letter is read as 0, which requests the URL continuously.

    Set an interval that suits the source

    A short delay against a public endpoint generates steady traffic and may trip rate limits. Size the interval to the rotation schedule you need to track.

  • A failed fetch is visible in the logs as a request through an internal proxy named <MOD-REMOTE-CRTSTORE>, with a status other than 200. See Check the fetch in the logs.

    Check after a rotation

    After the source rotates its keys, request a token signed with the new key and send it to the load balancer. If the source is an identity provider, request a fresh token from its token endpoint. If you run the issuer, mint one with the rotated private key. The example frontend returns 403 for a token it cannot verify, so a 403 means the load balancer did not accept the token.

    Check show ssl jwt first. If the new @<store>/<kid> entry is missing, the fetch has not picked the key up yet. A retired entry can linger in the same list until the load balancer reloads. If the new entry is present, check the algorithm and the certificate name that you pass to jwt_verify_cert.

  • The response body shares the load balancer’s buffer, so a large JWKS document or certificate chain may need a larger tune.bufsize value in the global section.

Limitations Jump to heading

  • Only JWKs that set a x5c field are inserted; an x5c field represents the public key in an X.509 certificate. A JWK that contains a bare public key isn’t supported, because a public key can’t be stored in a crt-store yet.
  • A JWK entry without a kid, and an entry whose x5c field can’t be decoded, will cause the whole response to be discarded, so none of the certificates in it are inserted.
  • A certificate whose kid disappears from the JWKS document stays in the store, and remains usable for validation, until the load balancer reloads. To retire a key safely, publish the new key alongside the old one and wait for the store to pick it up, keep both keys in the document until every token signed with the old key has expired, then remove the old key and reload the load balancer.
  • The module sends a plain GET request. It offers no arguments for a client certificate, an authorization header, or a custom CA file. For https URLs, the load balancer verifies the remote server with the httpclient.ssl.ca-file and httpclient.ssl.verify global settings.

Troubleshooting Jump to heading

Most problems with this module show up in the log line for the fetch or in the contents of the store. Check both, then work through the sections below for the usual causes.

Check the fetch in the logs Jump to heading

The module logs every fetch as an ordinary request through an internal proxy named <MOD-REMOTE-CRTSTORE>. A successful fetch and a failed one differ only in the HTTP status code and the termination state.

A fetch that returns 200:

successful fetch
text
-:- [23/Sep/2026:19:07:10.109] <MOD-REMOTE-CRTSTORE> -/- 2/0/1/5/6 200 8290 - - ---- 0/0/0/0/0 0/0 {} "GET http://127.0.0.1:8000/keys.jwk HTTP/1.1"
successful fetch
text
-:- [23/Sep/2026:19:07:10.109] <MOD-REMOTE-CRTSTORE> -/- 2/0/1/5/6 200 8290 - - ---- 0/0/0/0/0 0/0 {} "GET http://127.0.0.1:8000/keys.jwk HTTP/1.1"

A source that answers with an error, here a 404:

404 from the source
text
-:- [29/Sep/2026:15:55:27.321] <MOD-REMOTE-CRTSTORE> -/- 2/0/3/1/4 404 634 - - ---- 0/0/0/0/0 0/0 {} "GET http://127.0.0.1:8000/nosuchfile.jwk HTTP/1.1"
404 from the source
text
-:- [29/Sep/2026:15:55:27.321] <MOD-REMOTE-CRTSTORE> -/- 2/0/3/1/4 404 634 - - ---- 0/0/0/0/0 0/0 {} "GET http://127.0.0.1:8000/nosuchfile.jwk HTTP/1.1"

A source that can’t be reached, here because nothing listens on that port:

unreachable source
text
-:- [29/Sep/2026:15:54:46.653] <MOD-REMOTE-CRTSTORE> -/- 2/0/-1/-1/3023 503 0 - - SC-- 0/0/0/0/3 0/0 {} "GET http://127.0.0.1:8099/keys.jwk HTTP/1.1"
unreachable source
text
-:- [29/Sep/2026:15:54:46.653] <MOD-REMOTE-CRTSTORE> -/- 2/0/-1/-1/3023 503 0 - - SC-- 0/0/0/0/3 0/0 {} "GET http://127.0.0.1:8099/keys.jwk HTTP/1.1"

Read these fields from the log line:

  • <MOD-REMOTE-CRTSTORE> is the proxy name that the module logs its requests under. Filter the log on that name to see only the fetches.
  • The status code says whether the fetch worked. 200 means the module received a document, and any other status counts as a failure.
  • The termination state SC-- means the load balancer couldn’t connect to the source at all. It tells an unreachable server apart from one that answered with an error.
  • The bytes field is the response body plus about 190 bytes of headers. It can be lower than the full body when the load balancer stops reading after a non-200 status, so don’t use it to judge whether a document arrived complete.

A failed fetch produces no ALERT line and no message of its own, so the request line is the only record. An ALERT at startup reports a configuration error, such as a missing url argument. These lines appear only if the load balancer has a logger. To see them on the console, add a line such as log stderr format raw local0 info to the global section.

The store is empty Jump to heading

show ssl jwt lists the certificates in the store. When the command returns only the # filename header, nothing has been inserted. Work through these checks:

  • Check the fetch in the log. A status other than 200 means the source answered with an error or couldn’t be reached. See Check the fetch in the logs.
  • Check the timing arguments. delay and retry-delay take plain integers in seconds, so 1h is read as 1 second, and a value that starts with a letter is read as 0.
  • Wait for the first fetch. It happens soon after startup, but the store stays empty until a fetch succeeds, and a failed fetch is retried every retry-delay seconds.
  • Check that the document itself is usable. See The fetch returns 200 but nothing is inserted.

The fetch returns 200 but nothing is inserted Jump to heading

A 200 response that can’t be parsed is logged as an ordinary request line, with nothing to say that the module rejected the document, and the store keeps the certificates already in it. Common causes:

  • The document isn’t valid JSON. The packaged invalid.jwk file is an example of this, and you can use it to see the behavior.
  • A key in the document has no kid, or has an x5c field that can’t be decoded. Either one discards the whole response, so none of the certificates in it are inserted.
  • The document holds a bare public key rather than a certificate chain. Only entries with a full chain in the x5c field can be stored in a crt-store.
  • The response is larger than the load balancer’s buffer. A large JWKS document or certificate chain may need a larger tune.bufsize value in the global section.

Compare the store contents against the document when you suspect a parse problem. show ssl jwt lists one entry per key that was inserted, named @<store_name>/<kid>.

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page? Please leave feedback!

Previous page Verify crawlers Next page Device detection