Reference

trace

On this page

Available since

  • HAProxy 2.1
  • HAProxy Enterprise 2.1r1

Monitor load balancer events in real-time using a trace.

A trace can also be configured in the load balancer configuration file using a global trace directive.

Description Jump to heading

A trace captures events related to a particular source, such as HTTP, FastCGI or the Peers protocol. To begin a trace, follow these steps:

  1. Call the trace command to see available trace sources:

    bash
    echo "trace" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    output
    bash
    Supported trace sources and states (.=stopped, w=waiting, R=running) :
       [.] 0          : not a source, will immediately stop all traces
       [.] fcgi       : FastCGI multiplexer
       [.] h1         : HTTP/1 multiplexer
       [.] h2         : HTTP/2 multiplexer
       [.] stream     : Applicative stream
       [.] peers      : Peers protocol
    output
    bash
    Supported trace sources and states (.=stopped, w=waiting, R=running) :
       [.] 0          : not a source, will immediately stop all traces
       [.] fcgi       : FastCGI multiplexer
       [.] h1         : HTTP/1 multiplexer
       [.] h2         : HTTP/2 multiplexer
       [.] stream     : Applicative stream
       [.] peers      : Peers protocol

  2. Choose a source, such as h1 and check which events will be shown in the trace by adding the event parameter:

    bash
    echo "trace h1 event" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 event" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    output
    bash
    Supported events for source h1 (+=enabled, -=disabled):
       - none         : disable all event types
       - any          : enable all event types
       + h1c_new      : new H1 connection
       + h1c_recv     : Rx on H1 connection
       + h1c_send     : Tx on H1 connection
       + h1c_blk      : H1 connection blocked
       + h1c_wake     : H1 connection woken up
       + h1c_end      : H1 connection terminated
       + h1c_err      : error on H1 connection
       + rx_data      : receipt of any H1 data
       + rx_eoi       : receipt of end of H1 input
       + rx_headers   : receipt of H1 headers
       + rx_body      : receipt of H1 body
       + rx_trailerus : receipt of H1 trailers
       + tx_data      : transmission of any H1 data
       + tx_eoi       : transmission of end of H1 input
       + tx_headers   : transmission of all headers
       + tx_body      : transmission of H1 body
       + tx_trailerus : transmission of H1 trailers
       + h1s_new      : new H1 stream
       + h1s_blk      : H1 stream blocked
       + h1s_end      : H1 stream terminated
       + h1s_err      : error on H1 stream
       + strm_new     : app-layer stream creation
       + strm_recv    : receiving data for stream
       + strm_send    : sending data for stream
       + strm_wake    : stream woken up
       + strm_shut    : stream shutdown
       + strm_end     : detaching app-layer stream
       + strm_err     : stream error
    output
    bash
    Supported events for source h1 (+=enabled, -=disabled):
       - none         : disable all event types
       - any          : enable all event types
       + h1c_new      : new H1 connection
       + h1c_recv     : Rx on H1 connection
       + h1c_send     : Tx on H1 connection
       + h1c_blk      : H1 connection blocked
       + h1c_wake     : H1 connection woken up
       + h1c_end      : H1 connection terminated
       + h1c_err      : error on H1 connection
       + rx_data      : receipt of any H1 data
       + rx_eoi       : receipt of end of H1 input
       + rx_headers   : receipt of H1 headers
       + rx_body      : receipt of H1 body
       + rx_trailerus : receipt of H1 trailers
       + tx_data      : transmission of any H1 data
       + tx_eoi       : transmission of end of H1 input
       + tx_headers   : transmission of all headers
       + tx_body      : transmission of H1 body
       + tx_trailerus : transmission of H1 trailers
       + h1s_new      : new H1 stream
       + h1s_blk      : H1 stream blocked
       + h1s_end      : H1 stream terminated
       + h1s_err      : error on H1 stream
       + strm_new     : app-layer stream creation
       + strm_recv    : receiving data for stream
       + strm_send    : sending data for stream
       + strm_wake    : stream woken up
       + strm_shut    : stream shutdown
       + strm_end     : detaching app-layer stream
       + strm_err     : stream error

  3. Exclude specific events from the trace by prefixing the event name with a minus sign:

    bash
    echo "trace h1 event -h1c_new" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 event -h1c_new" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

    Or include an event by prefixing its name with a plus sign:

    bash
    echo "trace h1 event +h1c_new" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 event +h1c_new" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

    You can exclude all events by specifying the none event:

    bash
    echo "trace h1 event +none" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 event +none" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

    Or include all events by specifying the any event:

    bash
    echo "trace h1 event +any" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 event +any" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

  4. Set the trace level, which indicates the amount of detail to capture.

    bash
    echo "trace h1 level user" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 level user" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

    The level can be one of the following values:

    Level Description
    user Reports information suitable for observing live traffic, including HTTP requests and responses.
    proto Reports user-level information plus protocol-level updates such as HTTP headers and frame types.
    state Reports proto-level information plus transitions from one state to another.
    data Reports state-level information plus data transfers between layers.
    developer Reports everything including the names of functions that are invoked.

  5. Call trace with the verbosity parameter to see the current level of verbosity for the trace source:

    bash
    echo "trace h1 verbosity" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 verbosity" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    output
    bash
    Supported trace verbosities for source h1:
       quiet      : only report basic information with no decoding
       * clean      : only user-friendly stuff, generally suitable for level "user"
       minimal    : report only h1c/h1s state and flags, no real decoding
       simple     : add request/response status line or htx info when available
       advanced   : add header fields or frame decoding when available
       complete   : add full data dump when available
    output
    bash
    Supported trace verbosities for source h1:
       quiet      : only report basic information with no decoding
       * clean      : only user-friendly stuff, generally suitable for level "user"
       minimal    : report only h1c/h1s state and flags, no real decoding
       simple     : add request/response status line or htx info when available
       advanced   : add header fields or frame decoding when available
       complete   : add full data dump when available

    You can change the verbosity level as shown below:

    bash
    echo "trace h1 verbosity simple" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 verbosity simple" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

  6. Call trace with the lock parameter to see whether the current trace is locked onto any criteria:

    bash
    echo "trace h1 lock" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 lock" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    output
    bash
    Supported lock-on criteria for source h1:
       backend    : lock on the backend that started the trace
       connection : lock on the connection that started the trace
       frontend   : lock on the frontend that started the trace
       listener   : lock on the listener that started the trace
     * nothing    : do not lock on anything
       server     : lock on the server that started the trace
       session    : lock on the session that started the trace
       thread     : lock on the thread that started the trace
       h1s        : H1 stream
    output
    bash
    Supported lock-on criteria for source h1:
       backend    : lock on the backend that started the trace
       connection : lock on the connection that started the trace
       frontend   : lock on the frontend that started the trace
       listener   : lock on the listener that started the trace
     * nothing    : do not lock on anything
       server     : lock on the server that started the trace
       session    : lock on the session that started the trace
       thread     : lock on the thread that started the trace
       h1s        : H1 stream

    You can use the lock parameter to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers. Enable a lock on a particular criterion as shown below:

    bash
    echo "trace h1 lock connection" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 lock connection" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

  7. Call trace with the sink parameter to list available sinks, which are destinations where you can send the trace data:

    bash
    echo "trace h1 sink" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 sink" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    output
    bash
    Supported sinks for source h1 (*=current):
     * none       : no sink
       stdout     : standard output (fd#1)
       stderr     : standard output (fd#2)
       buf0       : in-memory ring buffer
    output
    bash
    Supported sinks for source h1 (*=current):
     * none       : no sink
       stdout     : standard output (fd#1)
       stderr     : standard output (fd#2)
       buf0       : in-memory ring buffer

    Specify a sink, such as stdout:

    bash
    echo "trace h1 sink stdout" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 sink stdout" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

  8. Call trace with the start now parameter to begin the trace:

    bash
    echo "trace h1 start now" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 start now" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

  9. When using the stdout sink, use journalctl to view trace logs:

    bash
    sudo journalctl --follow --unit hapee-2.8-lb
    bash
    sudo journalctl --follow --unit hapee-2.8-lb
    output
    bash
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)
    output
    bash
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)

  10. Stop the trace with the stop now parameter:

    bash
    echo "trace h1 stop now" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999
    bash
    echo "trace h1 stop now" | \
      sudo socat stdio tcp4-connect:127.0.0.1:9999

Examples Jump to heading

In the example below, we chain several commands to configure a trace of HTTP/1 requests:

bash
echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | \
  sudo socat stdio tcp4-connect:127.0.0.1:9999
bash
echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | \
  sudo socat stdio tcp4-connect:127.0.0.1:9999
bash
sudo journalctl --follow --unit hapee-2.8-lb
bash
sudo journalctl --follow --unit hapee-2.8-lb

See also Jump to heading

If this page was useful, please, Leave the feedback.

Previous page shutdown sessions server Next page update ssl ocsp-response
© 2023 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences