Reference
trace
Available since
- HAProxy 2.1
- HAProxy Enterprise 2.1r1
Monitor load balancer events in real-time using a trace.
A trace can also be configured in the load balancer configuration file using a global trace
directive.
Description Jump to heading
A trace captures events related to a particular source, such as HTTP, FastCGI or the Peers protocol. To begin a trace, follow these steps:
-
Call the
trace
command to see available trace sources:bashecho "trace" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace" | \sudo socat stdio tcp4-connect:127.0.0.1:9999outputbashSupported trace sources and states (.=stopped, w=waiting, R=running) :[.] 0 : not a source, will immediately stop all traces[.] fcgi : FastCGI multiplexer[.] h1 : HTTP/1 multiplexer[.] h2 : HTTP/2 multiplexer[.] stream : Applicative stream[.] peers : Peers protocoloutputbashSupported trace sources and states (.=stopped, w=waiting, R=running) :[.] 0 : not a source, will immediately stop all traces[.] fcgi : FastCGI multiplexer[.] h1 : HTTP/1 multiplexer[.] h2 : HTTP/2 multiplexer[.] stream : Applicative stream[.] peers : Peers protocol -
Choose a source, such as h1 and check which events will be shown in the trace by adding the
event
parameter:bashecho "trace h1 event" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 event" | \sudo socat stdio tcp4-connect:127.0.0.1:9999outputbashSupported events for source h1 (+=enabled, -=disabled):- none : disable all event types- any : enable all event types+ h1c_new : new H1 connection+ h1c_recv : Rx on H1 connection+ h1c_send : Tx on H1 connection+ h1c_blk : H1 connection blocked+ h1c_wake : H1 connection woken up+ h1c_end : H1 connection terminated+ h1c_err : error on H1 connection+ rx_data : receipt of any H1 data+ rx_eoi : receipt of end of H1 input+ rx_headers : receipt of H1 headers+ rx_body : receipt of H1 body+ rx_trailerus : receipt of H1 trailers+ tx_data : transmission of any H1 data+ tx_eoi : transmission of end of H1 input+ tx_headers : transmission of all headers+ tx_body : transmission of H1 body+ tx_trailerus : transmission of H1 trailers+ h1s_new : new H1 stream+ h1s_blk : H1 stream blocked+ h1s_end : H1 stream terminated+ h1s_err : error on H1 stream+ strm_new : app-layer stream creation+ strm_recv : receiving data for stream+ strm_send : sending data for stream+ strm_wake : stream woken up+ strm_shut : stream shutdown+ strm_end : detaching app-layer stream+ strm_err : stream erroroutputbashSupported events for source h1 (+=enabled, -=disabled):- none : disable all event types- any : enable all event types+ h1c_new : new H1 connection+ h1c_recv : Rx on H1 connection+ h1c_send : Tx on H1 connection+ h1c_blk : H1 connection blocked+ h1c_wake : H1 connection woken up+ h1c_end : H1 connection terminated+ h1c_err : error on H1 connection+ rx_data : receipt of any H1 data+ rx_eoi : receipt of end of H1 input+ rx_headers : receipt of H1 headers+ rx_body : receipt of H1 body+ rx_trailerus : receipt of H1 trailers+ tx_data : transmission of any H1 data+ tx_eoi : transmission of end of H1 input+ tx_headers : transmission of all headers+ tx_body : transmission of H1 body+ tx_trailerus : transmission of H1 trailers+ h1s_new : new H1 stream+ h1s_blk : H1 stream blocked+ h1s_end : H1 stream terminated+ h1s_err : error on H1 stream+ strm_new : app-layer stream creation+ strm_recv : receiving data for stream+ strm_send : sending data for stream+ strm_wake : stream woken up+ strm_shut : stream shutdown+ strm_end : detaching app-layer stream+ strm_err : stream error -
Exclude specific events from the trace by prefixing the event name with a minus sign:
bashecho "trace h1 event -h1c_new" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 event -h1c_new" | \sudo socat stdio tcp4-connect:127.0.0.1:9999Or include an event by prefixing its name with a plus sign:
bashecho "trace h1 event +h1c_new" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 event +h1c_new" | \sudo socat stdio tcp4-connect:127.0.0.1:9999You can exclude all events by specifying the none event:
bashecho "trace h1 event +none" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 event +none" | \sudo socat stdio tcp4-connect:127.0.0.1:9999Or include all events by specifying the any event:
bashecho "trace h1 event +any" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 event +any" | \sudo socat stdio tcp4-connect:127.0.0.1:9999 -
Set the trace level, which indicates the amount of detail to capture.
bashecho "trace h1 level user" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 level user" | \sudo socat stdio tcp4-connect:127.0.0.1:9999The level can be one of the following values:
Level Description user Reports information suitable for observing live traffic, including HTTP requests and responses. proto Reports user-level information plus protocol-level updates such as HTTP headers and frame types. state Reports proto-level information plus transitions from one state to another. data Reports state-level information plus data transfers between layers. developer Reports everything including the names of functions that are invoked. -
Call
trace
with theverbosity
parameter to see the current level of verbosity for the trace source:bashecho "trace h1 verbosity" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 verbosity" | \sudo socat stdio tcp4-connect:127.0.0.1:9999outputbashSupported trace verbosities for source h1:quiet : only report basic information with no decoding* clean : only user-friendly stuff, generally suitable for level "user"minimal : report only h1c/h1s state and flags, no real decodingsimple : add request/response status line or htx info when availableadvanced : add header fields or frame decoding when availablecomplete : add full data dump when availableoutputbashSupported trace verbosities for source h1:quiet : only report basic information with no decoding* clean : only user-friendly stuff, generally suitable for level "user"minimal : report only h1c/h1s state and flags, no real decodingsimple : add request/response status line or htx info when availableadvanced : add header fields or frame decoding when availablecomplete : add full data dump when availableYou can change the verbosity level as shown below:
bashecho "trace h1 verbosity simple" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 verbosity simple" | \sudo socat stdio tcp4-connect:127.0.0.1:9999 -
Call
trace
with thelock
parameter to see whether the current trace is locked onto any criteria:bashecho "trace h1 lock" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 lock" | \sudo socat stdio tcp4-connect:127.0.0.1:9999outputbashSupported lock-on criteria for source h1:backend : lock on the backend that started the traceconnection : lock on the connection that started the tracefrontend : lock on the frontend that started the tracelistener : lock on the listener that started the trace* nothing : do not lock on anythingserver : lock on the server that started the tracesession : lock on the session that started the tracethread : lock on the thread that started the traceh1s : H1 streamoutputbashSupported lock-on criteria for source h1:backend : lock on the backend that started the traceconnection : lock on the connection that started the tracefrontend : lock on the frontend that started the tracelistener : lock on the listener that started the trace* nothing : do not lock on anythingserver : lock on the server that started the tracesession : lock on the session that started the tracethread : lock on the thread that started the traceh1s : H1 streamYou can use the
lock
parameter to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers. Enable a lock on a particular criterion as shown below:bashecho "trace h1 lock connection" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 lock connection" | \sudo socat stdio tcp4-connect:127.0.0.1:9999 -
Call
trace
with thesink
parameter to list available sinks, which are destinations where you can send the trace data:bashecho "trace h1 sink" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 sink" | \sudo socat stdio tcp4-connect:127.0.0.1:9999outputbashSupported sinks for source h1 (*=current):* none : no sinkstdout : standard output (fd#1)stderr : standard output (fd#2)buf0 : in-memory ring bufferoutputbashSupported sinks for source h1 (*=current):* none : no sinkstdout : standard output (fd#1)stderr : standard output (fd#2)buf0 : in-memory ring bufferSpecify a sink, such as
stdout
:bashecho "trace h1 sink stdout" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 sink stdout" | \sudo socat stdio tcp4-connect:127.0.0.1:9999 -
Call
trace
with thestart now
parameter to begin the trace:bashecho "trace h1 start now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 start now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999 -
When using the
stdout
sink, usejournalctl
to view trace logs:bashsudo journalctl --follow --unit hapee-2.8-lbbashsudo journalctl --follow --unit hapee-2.8-lboutputbashFeb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)outputbashFeb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410) -
Stop the trace with the
stop now
parameter:bashecho "trace h1 stop now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999bashecho "trace h1 stop now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999
Examples Jump to heading
In the example below, we chain several commands to configure a trace of HTTP/1 requests:
bash
echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999
bash
echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | \sudo socat stdio tcp4-connect:127.0.0.1:9999
bash
sudo journalctl --follow --unit hapee-2.8-lb
bash
sudo journalctl --follow --unit hapee-2.8-lb
See also Jump to heading
If this page was useful, please, Leave the feedback.