HAProxy Enterprise Documentation 2.1r1

trace

Monitor load balancer events in real-time using a trace.

Description

A trace captures events related to a particular source, such as HTTP, FastCGI or the Peers protocol. To begin a trace, follow these steps:

  1. Call the trace command to see available trace sources:

    $ echo "trace" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
    
    Supported trace sources and states (.=stopped, w=waiting, R=running) :
     [.] 0          : not a source, will immediately stop all traces
     [.] fcgi       : FastCGI multiplexer
     [.] h1         : HTTP/1 multiplexer
     [.] h2         : HTTP/2 multiplexer
     [.] stream     : Applicative stream
     [.] peers      : Peers protocol
  2. Choose a source, such as h1 and check which events will be shown in the trace by adding the event parameter:

    $ echo "trace h1 event" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
    
    Supported events for source h1 (+=enabled, -=disabled):
      - none         : disable all event types
      - any          : enable all event types
      + h1c_new      : new H1 connection
      + h1c_recv     : Rx on H1 connection
      + h1c_send     : Tx on H1 connection
      + h1c_blk      : H1 connection blocked
      + h1c_wake     : H1 connection woken up
      + h1c_end      : H1 connection terminated
      + h1c_err      : error on H1 connection
      + rx_data      : receipt of any H1 data
      + rx_eoi       : receipt of end of H1 input
      + rx_headers   : receipt of H1 headers
      + rx_body      : receipt of H1 body
      + rx_trailerus : receipt of H1 trailers
      + tx_data      : transmission of any H1 data
      + tx_eoi       : transmission of end of H1 input
      + tx_headers   : transmission of all headers
      + tx_body      : transmission of H1 body
      + tx_trailerus : transmission of H1 trailers
      + h1s_new      : new H1 stream
      + h1s_blk      : H1 stream blocked
      + h1s_end      : H1 stream terminated
      + h1s_err      : error on H1 stream
      + strm_new     : app-layer stream creation
      + strm_recv    : receiving data for stream
      + strm_send    : sending data for stream
      + strm_wake    : stream woken up
      + strm_shut    : stream shutdown
      + strm_end     : detaching app-layer stream
      + strm_err     : stream error
  3. Exclude specific events from the trace by prefixing the event name with a minus sign:

    $ echo "trace h1 event -h1c_new" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock

    Or include an event by prefixing its name with a plus sign:

    $ echo "trace h1 event +h1c_new" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock

    You can exclude all events by specifying the none event:

    $ echo "trace h1 event +none" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock

    Or include all events by specifying the any event:

    $ echo "trace h1 event +any" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
  4. Set the trace level, which indicates the amount of detail to capture.

    $ echo "trace h1 level user" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock

    The level can be one of the following values:

    Level

    Description

    user

    Reports information suitable for observing live traffic, including HTTP requests and responses.

    proto

    Reports user-level information plus protocol-level updates such as HTTP headers and frame types.

    state

    Reports proto-level information plus transitions from one state to another.

    data

    Reports state-level information plus data transfers between layers.

    developer

    Reports everything including the names of functions that are invoked.

  5. Call trace with the verbosity parameter to see the current level of verbosity for the trace source:

    $ echo "trace h1 verbosity" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
    
    Supported trace verbosities for source h1:
        quiet      : only report basic information with no decoding
      * clean      : only user-friendly stuff, generally suitable for level "user"
        minimal    : report only h1c/h1s state and flags, no real decoding
        simple     : add request/response status line or htx info when available
        advanced   : add header fields or frame decoding when available
        complete   : add full data dump when available

    You can change the verbosity level as shown below:

    $ echo "trace h1 verbosity simple" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
  6. Call trace with the lock parameter to see whether the current trace is locked onto any criteria:

    $ echo "trace h1 lock" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
    
    Supported lock-on criteria for source h1:
        backend    : lock on the backend that started the trace
        connection : lock on the connection that started the trace
        frontend   : lock on the frontend that started the trace
        listener   : lock on the listener that started the trace
      * nothing    : do not lock on anything
        server     : lock on the server that started the trace
        session    : lock on the session that started the trace
        thread     : lock on the thread that started the trace
        h1s        : H1 stream

    You can use the lock parameter to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers. Enable a lock on a particular criterion as shown below:

    $ echo "trace h1 lock connection" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
  7. Call trace with the sink parameter to list available sinks, which are destinations where you can send the trace data:

    $ echo "trace h1 sink" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
    
    Supported sinks for source h1 (*=current):
      * none       : no sink
        stdout     : standard output (fd#1)
        stderr     : standard output (fd#2)
        buf0       : in-memory ring buffer

    Specify a sink, such as stdout:

    $ echo "trace h1 sink stdout" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
  8. Call trace with the start now parameter to begin the trace:

    $ echo "trace h1 start now" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
  9. When using the stdout sink, use journalctl to view trace logs:

    $ sudo journalctl --follow --unit hapee-2.1-lb
    
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1683] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00000010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 request fully xferred : [B] [MSG_DONE, MSG_RPBEFORE] - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1459] rcvd H1 response headers : [B] [MSG_DONE, MSG_DATA] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004010)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1492] H1 response fully rcvd : [B] [MSG_DONE, MSG_DONE] - "HTTP/1.1 200 OK" - h1c=0x55e4db986130(0x00000000) h1s=0x55e4db9869f0(0x00004410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1698] sending response headers : [F] [MSG_DONE, MSG_RPBEFORE] - "HTTP/1.1 200 OK" - h1c=0x55e4db9865b0(0x00000040) h1s=0x55e4db9862e0(0x00000410)
    Feb 18 22:05:02 lb1 hapee-lb[5076]: [00|h1|0|mux_h1.c:1955] H1 response fully xferred : [F] [MSG_DONE, MSG_DONE] - h1c=0x55e4db9865b0(0x00000000) h1s=0x55e4db9862e0(0x00004410)
  10. Stop the trace with the stop now parameter:

    $ echo "trace h1 stop now" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock

Examples

In the example below, we chain several commands to configure a trace of HTTP/1 requests:

$ echo "trace h1 event +any; trace h1 level user; trace h1 verbosity complete; trace h1 sink stdout; trace h1 start now" | sudo socat stdio /var/run/hapee-2.1/hapee-lb.sock
$ sudo journalctl --follow --unit hapee-2.1-lb

Next up

Lua scripts