HAProxy Enterprise Documentation 2.7r1

trace

Monitor load balancer events in real-time using a trace.

A trace can also be configured in the load balancer configuration file using a global trace directive.

Description

A trace captures events related to a particular source, such as HTTP, FastCGI or the Peers protocol. To begin a trace, follow these steps:

  1. Call the trace command to see available trace sources:

    $ echo "trace" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock
    Supported trace sources and states (.=stopped, w=waiting, R=running) :
 [.] 0          : not a source, will immediately stop all traces
 [.] fcgi       : FastCGI multiplexer
 [.] h1         : HTTP/1 multiplexer
 [.] h2         : HTTP/2 multiplexer
 [.] stream     : Applicative stream
 [.] peers      : Peers protocol

  2. Choose a source, such as h1 and check which events will be shown in the trace by adding the event parameter:

    $ echo "trace h1 event" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock
    Supported events for source h1 (+=enabled, -=disabled):
  - none         : disable all event types
  - any          : enable all event types
  + h1c_new      : new H1 connection
  + h1c_recv     : Rx on H1 connection
  + h1c_send     : Tx on H1 connection
  + h1c_blk      : H1 connection blocked
  + h1c_wake     : H1 connection woken up
  + h1c_end      : H1 connection terminated
  + h1c_err      : error on H1 connection
  + rx_data      : receipt of any H1 data
  + rx_eoi       : receipt of end of H1 input
  + rx_headers   : receipt of H1 headers
  + rx_body      : receipt of H1 body
  + rx_trailerus : receipt of H1 trailers
  + tx_data      : transmission of any H1 data
  + tx_eoi       : transmission of end of H1 input
  + tx_headers   : transmission of all headers
  + tx_body      : transmission of H1 body
  + tx_trailerus : transmission of H1 trailers
  + h1s_new      : new H1 stream
  + h1s_blk      : H1 stream blocked
  + h1s_end      : H1 stream terminated
  + h1s_err      : error on H1 stream
  + strm_new     : app-layer stream creation
  + strm_recv    : receiving data for stream
  + strm_send    : sending data for stream
  + strm_wake    : stream woken up
  + strm_shut    : stream shutdown
  + strm_end     : detaching app-layer stream
  + strm_err     : stream error

  3. Exclude specific events from the trace by prefixing the event name with a minus sign:

    $ echo "trace h1 event -h1c_new" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

    Or include an event by prefixing its name with a plus sign:

    $ echo "trace h1 event +h1c_new" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

    You can exclude all events by specifying the none event:

    $ echo "trace h1 event +none" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

    Or include all events by specifying the any event:

    $ echo "trace h1 event +any" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

  4. Set the trace level, which indicates the amount of detail to capture.

    $ echo "trace h1 level user" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

    The level can be one of the following values:

    Level

    Description

    user

    Reports information suitable for observing live traffic, including HTTP requests and responses.

    proto

    Reports user-level information plus protocol-level updates such as HTTP headers and frame types.

    state

    Reports proto-level information plus transitions from one state to another.

    data

    Reports state-level information plus data transfers between layers.

    developer

    Reports everything including the names of functions that are invoked.

  5. Call trace with the verbosity parameter to see the current level of verbosity for the trace source:

    $ echo "trace h1 verbosity" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock
    Supported trace verbosities for source h1:
    quiet      : only report basic information with no decoding
  * clean      : only user-friendly stuff, generally suitable for level "user"
    minimal    : report only h1c/h1s state and flags, no real decoding
    simple     : add request/response status line or htx info when available
    advanced   : add header fields or frame decoding when available
    complete   : add full data dump when available

    You can change the verbosity level as shown below:

    $ echo "trace h1 verbosity simple" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

  6. Call trace with the lock parameter to see whether the current trace is locked onto any criteria:

    $ echo "trace h1 lock" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock
    Supported lock-on criteria for source h1:
    backend    : lock on the backend that started the trace
    connection : lock on the connection that started the trace
    frontend   : lock on the frontend that started the trace
    listener   : lock on the listener that started the trace
  * nothing    : do not lock on anything
    server     : lock on the server that started the trace
    session    : lock on the session that started the trace
    thread     : lock on the thread that started the trace
    h1s        : H1 stream

    You can use the lock parameter to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers. Enable a lock on a particular criterion as shown below:

    $ echo "trace h1 lock connection" | |
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

  7. Call trace with the sink parameter to list available sinks, which are destinations where you can send the trace data:

    $ echo "trace h1 sink" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock
    Supported sinks for source h1 (*=current):
  * none       : no sink
    stdout     : standard output (fd#1)
    stderr     : standard output (fd#2)
    buf0       : in-memory ring buffer

    Specify a sink, such as stdout:

    $ echo "trace h1 sink stdout" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

  8. Call trace with the start now parameter to begin the trace:

    $ echo "trace h1 start now" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

  9. When using the stdout sink, use journalctl to view trace logs:

    $ sudo journalctl --follow --unit hapee-2.7-lb
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)
Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)

  10. Stop the trace with the stop now parameter:

    $ echo "trace h1 stop now" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

Examples

In the example below, we chain several commands to configure a trace of HTTP/1 requests:

$ echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | \
   sudo socat stdio unix-connect:/var/run/hapee-2.7/hapee-lb.sock

$ sudo journalctl --follow --unit hapee-2.7-lb

See also

Next up

user