Troubleshooting

Decrypt TLS traffic

On this page

This page applies to:

  • HAProxy Enterprise 3.0r1 and newer

When diagnosing network issues, you may need to analyze TLS-encrypted traffic to see the underlying application-layer protocol messages. You can use tcpdump to capture packets and save them to a .pcap, or packet capture file. You can then import such a file to Wireshark for analysis, but you must provide additional information to Wireshark so that it can decipher the traffic.

You can enable the logging of TLS keys in HAProxy Enterprise, which you can then import into Wireshark. Wireshark will use these secrets to decipher the encrypted packets in your .pcap file. As of version 3.0, you can produce a keylog file for both traffic between clients and the load balancer and traffic between the load balancer and backend servers.

Use for troubleshooting only

The following procedures will have you enable logging for TLS keys. There are both security and performance implications to consider when enabling logging for TLS keys. When you enable logging for keys, TLS secrets are logged in plaintext, which depending on your system may potentially be unsecure. Also, the load balancer will consume more memory per SSL session when this logging is enabled. Enable this behavior only while troubleshooting and be sure to secure your load balancer access logs.

Decrypt traffic between the load balancer and clients Jump to heading

To analyze TLS traffic between the load balancer and clients:

  1. Set these global options in the load balancer configuration:

    • tune.ssl.keylog to on. This activates the retrieval of the TLS keys you will use for decryption in Wireshark.
    • log should have a maximum length of at least 2048. If you’re using a remote rsyslog server, increase its $MaxMessageSize setting if needed.
    haproxy
    global
      tune.ssl.keylog on
      log 127.0.0.1 len 2048 local0
    haproxy
    global
      tune.ssl.keylog on
      log 127.0.0.1 len 2048 local0

  2. Force the load balancer and clients to use TLS 1.3 by adding the ssl-min-ver argument to your TLS bind line. TLS 1.3 is required for logging the TLS keys and for allowing you to decrypt the traffic in Wireshark:

    haproxy
    frontend  fe_main
      bind :443 ssl crt /etc/hapee-3.1/certs/cert.pem ssl-min-ver TLSv1.3
    haproxy
    frontend  fe_main
      bind :443 ssl crt /etc/hapee-3.1/certs/cert.pem ssl-min-ver TLSv1.3

    Tip

    You can also set the ssl-min-ver globally using the option ssl-default-bind-options. For example:

    haproxy
    global
      ssl-default-bind-options ssl-min-ver TLSv1.3
    haproxy
    global
      ssl-default-bind-options ssl-min-ver TLSv1.3

  3. Define a custom log format in your frontend that writes TLS session secrets to the access log. The log format uses sample fetches to retrieve the keys. We are using the frontend fetches here, as indicated by fc in the fetch names:

    haproxy
    frontend fe_main 
      log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_early_traffic_secret] \\CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_handshake_traffic_secret] \\SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_server_handshake_traffic_secret] \\CLIENT_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_client_traffic_secret_0] \\SERVER_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_server_traffic_secret_0] \\EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_exporter_secret] \\EARLY_EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_early_exporter_secret]"
    haproxy
    frontend fe_main 
      log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_early_traffic_secret] \\CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_handshake_traffic_secret] \\SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_server_handshake_traffic_secret] \\CLIENT_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_client_traffic_secret_0] \\SERVER_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_server_traffic_secret_0] \\EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_exporter_secret] \\EARLY_EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_early_exporter_secret]"

  4. Reload the load balancer to apply the configuration changes:

    nix
    sudo systemctl reload hapee-3.1-lb
    nix
    sudo systemctl reload hapee-3.1-lb

  5. Initiate a packet capture between the load balancer and clients using tcpdump to capture the traffic. For example, to save packets to a .pcap file on the load balancer instance named mycap.pcap, you can use the following command. Note that depending on your settings, you may need to change the port and network interface (-i). The port is the port on which clients make TLS connection to your load balancer.

    nix
    sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap
    nix
    sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

    Tip

    Depending on your OS, you can list your network interfaces using a command such as ifconfig -a or ip link show.

  6. While your capture is running, and after a client connects to the load balancer, the access log will include the keys for the TLS session:

    text
    Aug 13 14:22:09 localhost hapee-lb[13604]: 192.168.50.1:61186 [13/Aug/2025:14:22:09.034] fe_main~ webservers/s1 0/0/29/72/101 200 858 - - ---- 1/1/0/0/0 0/0 "GET https://192.168.50.25/ HTTP/2.0" CLIENT_EARLY_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A - \CLIENT_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A c0022ed991a22988b68694a159aa870 \SERVER_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A 2f2ea07d5adc677975056cfa935b8f3 \CLIENT_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 2cc50910694380586145693fcf59eaa \SERVER_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 6d5fc6cb85c475affcc2f77df76b4ef \EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A e3fc120e962c86421ef8ff39e0d9a382 \EARLY_EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -
    text
    Aug 13 14:22:09 localhost hapee-lb[13604]: 192.168.50.1:61186 [13/Aug/2025:14:22:09.034] fe_main~ webservers/s1 0/0/29/72/101 200 858 - - ---- 1/1/0/0/0 0/0 "GET https://192.168.50.25/ HTTP/2.0" CLIENT_EARLY_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A - \CLIENT_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A c0022ed991a22988b68694a159aa870 \SERVER_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A 2f2ea07d5adc677975056cfa935b8f3 \CLIENT_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 2cc50910694380586145693fcf59eaa \SERVER_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 6d5fc6cb85c475affcc2f77df76b4ef \EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A e3fc120e962c86421ef8ff39e0d9a382 \EARLY_EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -

    Note that each TLS session (connection) will generate its own keys.

  7. Save a log line that pertains to the client in question from the access log to a text file, such as request.txt. Then, use sed to replace the backslashes with newlines and remove the preceding text you don’t need:

    nix
    cat request.txt | sed -rn 's/(^.*)(CLIENT_EARLY_TRAFFIC_SECRET.*)$/\2/p' | sed 's/ \\/\n/g' > keylog.txt
    nix
    cat request.txt | sed -rn 's/(^.*)(CLIENT_EARLY_TRAFFIC_SECRET.*)$/\2/p' | sed 's/ \\/\n/g' > keylog.txt

    The result will be a file containing only the keys and values.

    keylog.txt
    text
    CLIENT_EARLY_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -
    CLIENT_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A c0022ed991a22988b68694a159aa87
    SERVER_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A 2f2ea07d5adc677975056cfa935b8f
    CLIENT_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 2cc50910694380586145693fcf59ea
    SERVER_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 6d5fc6cb85c475affcc2f77df76b4e
    EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A e3fc120e962c86421ef8ff39e0d9a38
    EARLY_EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -
    keylog.txt
    text
    CLIENT_EARLY_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -
    CLIENT_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A c0022ed991a22988b68694a159aa87
    SERVER_HANDSHAKE_TRAFFIC_SECRET 8D7D9AC099BC0B0F67EABE8AC587A 2f2ea07d5adc677975056cfa935b8f
    CLIENT_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 2cc50910694380586145693fcf59ea
    SERVER_TRAFFIC_SECRET_0 8D7D9AC099BC0B0F67EABE8AC587A 6d5fc6cb85c475affcc2f77df76b4e
    EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A e3fc120e962c86421ef8ff39e0d9a38
    EARLY_EXPORTER_SECRET 8D7D9AC099BC0B0F67EABE8AC587A -

  8. Import this file into Wireshark via Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

  9. Open the .pcap file with your captured traffic in Wireshark to see the deciphered traffic.

Once you have finished troubleshooting:

  1. In your load balancer configuration, set tune.ssl.keylog to off in the global section, or delete the line entirely. This disables the logging of TLS keys.

  2. Remove the log format line that retrieves the TLS keys from your configuration.

  3. Reload the load balancer to apply the configuration changes:

    nix
    sudo systemctl reload hapee-3.1-lb
    nix
    sudo systemctl reload hapee-3.1-lb

Decrypt traffic between the load balancer and backend servers Jump to heading

To analyze TLS traffic between the load balancer and backend servers:

  1. Set these global options in the load balancer configuration:

    • tune.ssl.keylog to on. This activates the retrieval of the TLS keys you will use for decryption in Wireshark.
    • log should have a maximum length of at least 2048. If you’re using a remote rsyslog server, increase its $MaxMessageSize setting if needed.
    haproxy
    global
      tune.ssl.keylog on
      log 127.0.0.1 len 2048 local0
    haproxy
    global
      tune.ssl.keylog on
      log 127.0.0.1 len 2048 local0

  2. Force the load balancer and the backend servers to use TLS 1.3 by adding the ssl-min-ver argument to the servers. TLS 1.3 is required for logging the TLS keys and for allowing you to decrypt the traffic in Wireshark:

    haproxy
    backend servers
      server s1 192.168.56.50:443 ssl verify required ca-file /etc/haproxy/certs/ca.crt ssl-min-ver TLSv1.3
    haproxy
    backend servers
      server s1 192.168.56.50:443 ssl verify required ca-file /etc/haproxy/certs/ca.crt ssl-min-ver TLSv1.3

    Note that here we have also added verify required to our server line and have provided the CA certificate using ca-file. This enforces a check where the load balancer will verify the server certificate. For more information see verify reference.

    Tip

    You can also set the ssl-min-ver globally using the global option ssl-default-server-options. For example:

    haproxy
    global
      ssl-default-server-options ssl-min-ver TLSv1.3
    haproxy
    global
      ssl-default-server-options ssl-min-ver TLSv1.3

  3. Define a custom log format in your frontend that writes TLS session secrets to the access log. The log format uses sample fetches to retrieve the keys. We are using the backend fetches here, as indicated by bc in the fetch names:

    haproxy
    frontend fe_main
      log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret] \\CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret] \\SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret] \\CLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0] \\SERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0] \\EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret] \\EARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"
    haproxy
    frontend fe_main
      log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret] \\CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret] \\SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret] \\CLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0] \\SERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0] \\EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret] \\EARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"

  4. Reload the load balancer to apply the configuration changes:

    nix
    sudo systemctl reload hapee-3.1-lb
    nix
    sudo systemctl reload hapee-3.1-lb

  5. Initiate a packet capture between the load balancer and the backend servers using tcpdump to capture the traffic. For example, to save packets to a .pcap file on the load balancer instance named mycap.pcap, you could use the following command. Note that you may need to change the port and network interface (-i) depending on your settings. The port is the port on which your load balancer connects to your backend servers.

    nix
    sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap
    nix
    sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

    Tip

    You can list your network interfaces using a command such as ifconfig -a or ip link show, depending on your OS.

  6. While your capture is running, and after a client connects to the load balancer, the access log will include the keys for the TLS session:

    text
    Aug 13 14:22:09 localhost hapee-lb[13604]: 192.168.50.1:61186 [13/Aug/2025:14:22:09.034] fe_main~ webservers/s1 0/0/29/72/101 200 858 - - ---- 1/1/0/0/0 0/0 "GET https://192.168.50.25/ HTTP/2.0" CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 - \CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 15ab9abf57145fe49c73d9a617eca \SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 09bded135c6b85959d0c2eaf09d177 \CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 155b07c8fcef945cbad456f6b11e21 \SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 18ed3dc1188b7ed1085cbdf41b0f03 \EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 9479651fd91e38d549b284ecae7c64 \EARLY_EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 -
    text
    Aug 13 14:22:09 localhost hapee-lb[13604]: 192.168.50.1:61186 [13/Aug/2025:14:22:09.034] fe_main~ webservers/s1 0/0/29/72/101 200 858 - - ---- 1/1/0/0/0 0/0 "GET https://192.168.50.25/ HTTP/2.0" CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 - \CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 15ab9abf57145fe49c73d9a617eca \SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 09bded135c6b85959d0c2eaf09d177 \CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 155b07c8fcef945cbad456f6b11e21 \SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 18ed3dc1188b7ed1085cbdf41b0f03 \EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 9479651fd91e38d549b284ecae7c64 \EARLY_EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 -

    Note that each TLS session (connection) will generate its own keys.

  7. Save a log line that pertains to the client in question from the access log to a text file, such as request.txt. Then, use sed to replace the backslashes with newlines and remove the preceding text you don’t need:

    nix
    cat request.txt | sed -rn 's/(^.*)(CLIENT_EARLY_TRAFFIC_SECRET.*)$/\2/p' | sed 's/ \\/\n/g' > keylog.txt
    nix
    cat request.txt | sed -rn 's/(^.*)(CLIENT_EARLY_TRAFFIC_SECRET.*)$/\2/p' | sed 's/ \\/\n/g' > keylog.txt

    The result will be a file containing only the keys and values.

    keylog.txt
    text
    CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 -
    CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 15ab9abf57145fe49c73d9a617eca
    SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 09bded135c6b85959d0c2eaf09d177
    CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 155b07c8fcef945cbad456f6b11e21
    SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 18ed3dc1188b7ed1085cbdf41b0f03
    EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 9479651fd91e38d549b284ecae7c64
    EARLY_EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 -
    keylog.txt
    text
    CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 -
    CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 15ab9abf57145fe49c73d9a617eca
    SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E2 09bded135c6b85959d0c2eaf09d177
    CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 155b07c8fcef945cbad456f6b11e21
    SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E2 18ed3dc1188b7ed1085cbdf41b0f03
    EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 9479651fd91e38d549b284ecae7c64
    EARLY_EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E2 -

    Note that each TLS session (connection) will generate its own keys.

  8. Save the lines from the access logs containing the secrets to a text file. Import the file into Wireshark via Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

  9. Open the .pcap file with your captured traffic in Wireshark to see the deciphered traffic.

Once you have finished troubleshooting:

  1. In your load balancer configuration, set tune.ssl.keylog to off in the global section, or delete the line entirely. This disables the logging of TLS keys.

  2. Remove the log format line that retrieves the TLS keys from your configuration.

  3. Reload the load balancer to apply the configuration changes:

    nix
    sudo systemctl reload hapee-3.1-lb
    nix
    sudo systemctl reload hapee-3.1-lb

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?

Previous page Contact Us Next page Enable core dumps
© 2025 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences