Troubleshooting

Debugging

This section describes how to use debugging features.

Trace a source in real time Jump to heading

Use a trace to monitor load balancer events in real time. A trace captures events related to a particular source such as HTTP, FastCGI, or the Peers protocol.

The trace command is already supported in the Runtime API. Version 2.7 adds experimental support for an analogous trace directive in the load balancer configuration file.

To define a trace in the configuration file, add the expose-experimental-directives directive followed by the trace directive in the global section.

haproxy
global
expose-experimental-directives
trace h1 event +any
haproxy
global
expose-experimental-directives
trace h1 event +any

To review the events, use journalctl.

bash
sudo journalctl --follow --unit hapee-2.8-lb | head -3
bash
sudo journalctl --follow --unit hapee-2.8-lb | head -3
output
bash
-- Logs begin at Thu 2023-02-23 18:51:08 UTC. --
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers :
[B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)
res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200)
conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001)
sc=0x55ec8c2af5f0(0x00000011)
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred :
[B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)
res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200)
conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001)
sc=0x55ec8c2af5f0(0x00000011)
output
bash
-- Logs begin at Thu 2023-02-23 18:51:08 UTC. --
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers :
[B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)
res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200)
conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001)
sc=0x55ec8c2af5f0(0x00000011)
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred :
[B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)
res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200)
conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001)
sc=0x55ec8c2af5f0(0x00000011)

You can also use the show events Runtime API command to display the trace events.

bash
echo "show events buf0" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock | head -3
bash
echo "show events buf0" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock | head -3
output
bash
<0>2021-02-22T16:44:21.014274+00:00 [00|h1|0|mux_h1.c:1459] rcvd H1 request headers :
[F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000)
h1s=0x5617388cd620(0x00000010)
<0>2021-02-22T16:44:21.017431+00:00 [00|h1|0|mux_h1.c:1492] H1 request fully rcvd :
[F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000)
h1s=0x5617388cd620(0x00000410)
output
bash
<0>2021-02-22T16:44:21.014274+00:00 [00|h1|0|mux_h1.c:1459] rcvd H1 request headers :
[F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000)
h1s=0x5617388cd620(0x00000010)
<0>2021-02-22T16:44:21.017431+00:00 [00|h1|0|mux_h1.c:1492] H1 request fully rcvd :
[F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000)
h1s=0x5617388cd620(0x00000410)

To display the trace definition, use the show trace Runtime API command.

bash
echo "show trace" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock
bash
echo "show trace" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock
output
bash
Supported trace sources and states (.=stopped, w=waiting, R=running) :
[.] h2 -> none [drp 0] [HTTP/2 multiplexer]
[.] fcgi -> none [drp 0] [FastCGI multiplexer]
[R] h1 -> stdout [drp 0] [HTTP/1 multiplexer]
[.] stream -> none [drp 0] [Applicative stream]
[.] peers -> none [drp 0] [Peers protocol]
[.] check -> none [drp 0] [Health-check]
[.] pt -> none [drp 0] [Passthrough multiplexer]
output
bash
Supported trace sources and states (.=stopped, w=waiting, R=running) :
[.] h2 -> none [drp 0] [HTTP/2 multiplexer]
[.] fcgi -> none [drp 0] [FastCGI multiplexer]
[R] h1 -> stdout [drp 0] [HTTP/1 multiplexer]
[.] stream -> none [drp 0] [Applicative stream]
[.] peers -> none [drp 0] [Peers protocol]
[.] check -> none [drp 0] [Health-check]
[.] pt -> none [drp 0] [Passthrough multiplexer]

Anonymize debug output Jump to heading

You can anonymize the output of some Runtime API commands that might otherwise reveal sensitive or confidential data such as IP addresses, names of sections, and hostnames.

In anonymization, sensitive fields are replaced with 24-bit hashes that preserve sufficient consistency between elements to help troubleshooters identify relationships between elements.

To anonymize a Runtime API command, precede it with set anon on;.

bash
echo "set anon on; show sess" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock
bash
echo "set anon on; show sess" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock

The hash key is a 32-bit number between 0 and 4294967295, where 0 disables the anonymizing function. The following methods are supported for setting the hash key:

  • Specify a key at each invocation: set anon on <key>;

  • Specify a global hash key for all invocations: set anon global-key <key> CLI command or anonkey global configuration directive

  • Omit a key and allow to generate a random key

    bash
    echo "set anon global-key 123456789; set anon on; show sess" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock
    bash
    echo "set anon global-key 123456789; set anon on; show sess" | sudo socat stdio /var/run/hapee-2.8/hapee-lb.sock

Anonymize configuration file Jump to heading

To dump an anonymized copy of the configuration file with IP addresses, section names, and hostnames masked, use the hapee-lb command with the -dC<key> flag, specifying an arbitrary number as the hash key.

bash
/opt/hapee-2.7/sbin/hapee-lb -dC123456789 -f /etc/hapee-2.8/hapee-lb.cfg
bash
/opt/hapee-2.7/sbin/hapee-lb -dC123456789 -f /etc/hapee-2.8/hapee-lb.cfg

The key generates hashed values in the same way as the set anon CLI command, allowing you to correlate output generated by the two methods when troubleshooting.

See also Jump to heading

If this page was useful, please, Leave the feedback.