HAProxy Enterprise Documentation 2.3r1

trace

Monitor load balancer events in real-time using a trace.

Description

A trace captures events related to a particular source, such as HTTP, FastCGI or the Peers protocol. To begin a trace, follow these steps:

Call the trace command to see available trace sources:

$ echo "trace" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Supported trace sources and states (.=stopped, w=waiting, R=running) :
 [.] 0          : not a source, will immediately stop all traces
 [.] fcgi       : FastCGI multiplexer
 [.] h1         : HTTP/1 multiplexer
 [.] h2         : HTTP/2 multiplexer
 [.] stream     : Applicative stream
 [.] peers      : Peers protocol

Choose a source, such as h1 and check which events will be shown in the trace by adding the event parameter:

$ echo "trace h1 event" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Supported events for source h1 (+=enabled, -=disabled):
  - none         : disable all event types
  - any          : enable all event types
  + h1c_new      : new H1 connection
  + h1c_recv     : Rx on H1 connection
  + h1c_send     : Tx on H1 connection
  + h1c_blk      : H1 connection blocked
  + h1c_wake     : H1 connection woken up
  + h1c_end      : H1 connection terminated
  + h1c_err      : error on H1 connection
  + rx_data      : receipt of any H1 data
  + rx_eoi       : receipt of end of H1 input
  + rx_headers   : receipt of H1 headers
  + rx_body      : receipt of H1 body
  + rx_trailerus : receipt of H1 trailers
  + tx_data      : transmission of any H1 data
  + tx_eoi       : transmission of end of H1 input
  + tx_headers   : transmission of all headers
  + tx_body      : transmission of H1 body
  + tx_trailerus : transmission of H1 trailers
  + h1s_new      : new H1 stream
  + h1s_blk      : H1 stream blocked
  + h1s_end      : H1 stream terminated
  + h1s_err      : error on H1 stream
  + strm_new     : app-layer stream creation
  + strm_recv    : receiving data for stream
  + strm_send    : sending data for stream
  + strm_wake    : stream woken up
  + strm_shut    : stream shutdown
  + strm_end     : detaching app-layer stream
  + strm_err     : stream error

Exclude specific events from the trace by prefixing the event name with a minus sign:

$ echo "trace h1 event -h1c_new" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Or include an event by prefixing its name with a plus sign:

$ echo "trace h1 event +h1c_new" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

You can exclude all events by specifying the none event:

$ echo "trace h1 event +none" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Or include all events by specifying the any event:

$ echo "trace h1 event +any" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Set the trace level, which indicates the amount of detail to capture.

$ echo "trace h1 level user" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

The level can be one of the following values:

Level	Description
user	Reports information suitable for observing live traffic, including HTTP requests and responses.
proto	Reports user-level information plus protocol-level updates such as HTTP headers and frame types.
state	Reports proto-level information plus transitions from one state to another.
data	Reports state-level information plus data transfers between layers.
developer	Reports everything including the names of functions that are invoked.

Call trace with the verbosity parameter to see the current level of verbosity for the trace source:

$ echo "trace h1 verbosity" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Supported trace verbosities for source h1:
    quiet      : only report basic information with no decoding
  * clean      : only user-friendly stuff, generally suitable for level "user"
    minimal    : report only h1c/h1s state and flags, no real decoding
    simple     : add request/response status line or htx info when available
    advanced   : add header fields or frame decoding when available
    complete   : add full data dump when available

You can change the verbosity level as shown below:

$ echo "trace h1 verbosity simple" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Call trace with the lock parameter to see whether the current trace is locked onto any criteria:

$ echo "trace h1 lock" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Supported lock-on criteria for source h1:
    backend    : lock on the backend that started the trace
    connection : lock on the connection that started the trace
    frontend   : lock on the frontend that started the trace
    listener   : lock on the listener that started the trace
  * nothing    : do not lock on anything
    server     : lock on the server that started the trace
    session    : lock on the session that started the trace
    thread     : lock on the thread that started the trace
    h1s        : H1 stream

You can use the lock parameter to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers. Enable a lock on a particular criterion as shown below:

$ echo "trace h1 lock connection" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Call trace with the sink parameter to list available sinks, which are destinations where you can send the trace data:

$ echo "trace h1 sink" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Supported sinks for source h1 (*=current):
  * none       : no sink
    stdout     : standard output (fd#1)
    stderr     : standard output (fd#2)
    buf0       : in-memory ring buffer

Specify a sink, such as stdout:

$ echo "trace h1 sink stdout" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Call trace with the start now parameter to begin the trace:

$ echo "trace h1 start now" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

When using the stdout sink, use journalctl to view trace logs:

$ sudo journalctl --follow --unit hapee-2.3-lb

Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)

Stop the trace with the stop now parameter:

$ echo "trace h1 stop now" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock

Examples

In the example below, we chain several commands to configure a trace of HTTP/1 requests:

$ echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | sudo socat stdio /var/run/hapee-2.3/hapee-lb.sock
$ sudo journalctl --follow --unit hapee-2.3-lb

More Versions

HAProxy Products Documentation

trace

Description

Examples

See also

Your feedback is important to us!