Administration

Manage HAProxy Enterprise logs

HAProxy Enterprise generates two types of logs: access logs, also called traffic logs, and administrative logs.

Enable access logs Jump to heading

Access logs, also called traffic logs, record information about client connections and requests. The logs are stored in the /var/log/hapee-<VERSION> directory and have file names prefixed with lb-access- plus a timestamp.

To enable access logs:

  1. Add the following log directive in the global section of the HAProxy Enterprise configuration file, if one does not already exist:

    haproxy
    global
    log 127.0.0.1 local0
    haproxy
    global
    log 127.0.0.1 local0

    This definition, which does not set a severity level for events, captures all events and tags them with the local0 syslog facility code. An rsyslog configuration file at /etc/rsyslog.d/hapee-<VERSION>-lb.conf routes these events to the access log file.

    You can change the IP address to send logs to a remote rsyslog server over UDP.

  2. In your frontend or defaults section, add a log global directive, which applies the log directive from the global section to the frontend.

    haproxy
    frontend fe_main
    log global
    haproxy
    frontend fe_main
    log global

Enable administrative logs Jump to heading

Administrative logs record information about load balancer process events such as when the load balancer starts or stops. The logs are stored in the /var/log/hapee-<VERSION> directory and have file names prefixed with lb-admin- plus a timestamp.

To enable administrative logs:

  1. Add the following log directive in the global section of the HAProxy Enterprise configuration file, if one does not already exist:

    haproxy
    global
    log 127.0.0.1 local1 notice
    haproxy
    global
    log 127.0.0.1 local1 notice

    This definition, which sets its severity level to notice, captures most log events but omits the more verbose traffic events captured by the access logs. It tags those events with the syslog facility code local1. An rsyslog configuration file at /etc/rsyslog.d/hapee-<VERSION>-lb.conf routes these events to the admin log file. Basically, they have the same contents as access logs but without the info level details about connection and request events.

    You can change the IP address to send logs to a remote rsyslog server over UDP.

  2. In your frontend or defaults section, add a log global directive, which applies the log directive from the global section to the frontend.

    haproxy
    frontend fe_main
    log global
    haproxy
    frontend fe_main
    log global

Set logging levels Jump to heading

Log levels are the same as the severity levels used by the syslog service. Below, we set the level to notice, which captures all events at that level and above (up to emerg):

haproxy
global
log 127.0.0.1 local1 notice
haproxy
global
log 127.0.0.1 local1 notice

HAProxy Enterprise tags each loggable event with a severity level. For example, it categorizes log messages related to connections and HTTP requests with the info severity level. Other events are categorized using one of the other, less verbose levels. From most to least verbose, the severity levels are:

Level Events at this level
emerg Errors such as running out of operating system file descriptors.
alert Some rare cases where something unexpected has happened, such as being unable to cache a response.
crit Not used.
err Errors such as being unable to parse a map file, being unable to parse the HAProxy configuration file, and when an operation on a stick table fails.
warning Certain important, but non-critical, errors such as failing to set a request header or failing to connect to a DNS nameserver.
notice Changes to a server’s state, such as being UP or DOWN or when a server is disabled. Other events at startup, such as starting proxies and loading modules, are also included. Health check logging, if enabled, also uses this level.
info TCP connection and HTTP request details and errors.
debug You may write custom Lua code that logs at this level.

Set conditional log levels Jump to heading

Sometimes, you’ll want to capture a more verbose log entry for a particular request or response, such as when the server returned an error. To change the log level for individual requests or responses that match specified criteria, use one of these directives, depending on whether you’re targeting the request phase or response phase, and whether your’re in HTTP or TCP mode:

  • http-request set-log-level <level> [ { if | unless } <condition> ]
  • http-response set-log-level <level> [ { if | unless } <condition> ]
  • tcp-request content set-log-level <level> [ { if | unless } <condition> ]
  • tcp-response content set-log-level <level> [ { if | unless } <condition> ]

Using these directives, you can set the log level to any of the eight syslog levels or to the special level silent, which disables logging for this request/response. This rule is not final so the last matching rule wins.

Specify a condition using the ACL syntax:

haproxy
frontend www
bind :80
acl failed_request status 400 401 403 404 405 408 429 500 503
http-response set-log-level err if failed_request
haproxy
frontend www
bind :80
acl failed_request status 400 401 403 404 405 408 429 500 503
http-response set-log-level err if failed_request

Log ASAP Jump to heading

By default, logs are emitted when the session terminates. That is, when all log values are finalized. Capturing some log values, such as total transfer time and message size, can delay generation of the log content, particularly for a large file transfer.

To avoid such delays, you can choose to log available values as quickly as possible by specifying the logasap option. With this option, the load balancer creates the log message as soon as the server connection has been established in mode tcp or as soon as the server sends the complete headers in mode http.

Values not finalized at this time are total time and total size metrics, and they are prefixed with a plus sign “+” in the log to indicate that they are not true, final values.

Enable early logging by adding option logasap to the defaults or frontend section, as shown below. In this example, we also capture the Content-Length header in the logs so that the logs at least indicate how many bytes are expected to be transferred.

haproxy
frontend fe_main
bind 0.0.0.0:80
mode http
log global
option httplog
option logasap
http-request capture req.hdr(Content-Length) len 15
haproxy
frontend fe_main
bind 0.0.0.0:80
mode http
log global
option httplog
option logasap
http-request capture req.hdr(Content-Length) len 15
output
text
>>> Feb 6 12:14:14 localhost \
hapee-lb[14389]: 10.0.1.2:33317 [06/Feb/2019:12:14:14.655] fe_main \
static/srv1 9/10/7/14/+30 200 +243 - - ---- 3/1/1/1/0 1/0 \
"GET /image.iso HTTP/1.0"
output
text
>>> Feb 6 12:14:14 localhost \
hapee-lb[14389]: 10.0.1.2:33317 [06/Feb/2019:12:14:14.655] fe_main \
static/srv1 9/10/7/14/+30 200 +243 - - ---- 3/1/1/1/0 1/0 \
"GET /image.iso HTTP/1.0"

Set the log format Jump to heading

By default, HAProxy Enterprise logs are sparse, capturing minimal information about each TCP connection in the access logs:

text
>>> Jan 27 21:15:06 localhost hapee-lb[2901]: Connect from 192.168.50.1:61459 to 192.168.50.25:80 \
(fe_main/HTTP)
text
>>> Jan 27 21:15:06 localhost hapee-lb[2901]: Connect from 192.168.50.1:61459 to 192.168.50.25:80 \
(fe_main/HTTP)

The information includes:

Field Example value
the PID of the HAProxy Enterprise process hapee-lb[2901]
the text “Connect from” followed by the client’s source IP address and port Connect from 192.168.50.1:61459
the text “to” followed by the destination IP address and port to 192.168.50.25:80
the name of the frontend that received the request and the mode (e.g. HTTP) (fe_main/HTTP)

To get a more detailed log, use one of the directives describes in the following sections.

TCP log format Jump to heading

The option tcplog directive is suitable for frontend sections that either specify mode tcp or don’t set mode at all.

haproxy
defaults
mode tcp
option tcplog
haproxy
defaults
mode tcp
option tcplog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Jan 27 21:22:53 localhost hapee-lb[3042]: 192.168.50.1:61535 [27/Jan/2021:21:22:10.944] \
fe_main be_servers/s1 1/0/42404 2895 cD 1/1/0/0/0 0/0
output
text
>>> Jan 27 21:22:53 localhost hapee-lb[3042]: 192.168.50.1:61535 [27/Jan/2021:21:22:10.944] \
fe_main be_servers/s1 1/0/42404 2895 cD 1/1/0/0/0 0/0

The information included is detailed in the TCP log format section of the reference guide.

The tcplog log format is equivalent to the following log-format definition, if you were to set it yourself:

haproxy
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
haproxy
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

HTTP log format Jump to heading

The option httplog directive is suitable for frontend sections that specify mode http. This option provides the same information as the tcplog option, but with additional HTTP details.

haproxy
defaults
mode http
option httplog
haproxy
defaults
mode http
option httplog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Jan 27 21:52:56 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} "GET / HTTP/1.1"
output
text
>>> Jan 27 21:52:56 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} "GET / HTTP/1.1"

The information included is detailed in the HTTP log format section of the reference guide.

The httplog log format is equivalent to the following log-format definition, if you were to set it yourself:

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

CLF log format Jump to heading

If you intend to pass the HTTP log to a log analyzer that supports the CLF format, you can specify the optional clf argument:

haproxy
defaults
mode http
option httplog clf
haproxy
defaults
mode http
option httplog clf

The CLF format provides the same details as the HTTP log format, but in the order required for CLF parsing. It is equivalent to the following log-format definition:

haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"
haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"

HTTPS log format Jump to heading

Available since

  • HAProxy Enterprise 2.5r1

The option httpslog directive is suitable for HTTP over TLS connections. This option provides the same information as the option httplog directive, but with additional TLS details.

haproxy
defaults
mode http
option httpslog
haproxy
defaults
mode http
option httpslog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Feb 6 12:14:14 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} \
"GET / HTTP/1.1" 0/0/0/0/0 1wt.eu/TLSv1.3/TLS_AES_256_GCM_SHA384
output
text
>>> Feb 6 12:14:14 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} \
"GET / HTTP/1.1" 0/0/0/0/0 1wt.eu/TLSv1.3/TLS_AES_256_GCM_SHA384

The information included is detailed in the HTTPS log format section of the reference guide.

The httpslog log format is equivalent to the following log-format definition:

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"
haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

Custom log format Jump to heading

You can define your own log format and record a custom set of information about connections or HTTP requests. Your log format can include variables and values from fetch methods.

Use the log-format directive to create a new log format in your defaults or frontend section. It specifies a string that contains variables referring to the fields that you’d like to capture.

Often, you will want to keep the existing information collected by the premade log format rules and append new information to them. From version 2.7r1 onward, to make that easier, you can use one of the following environment variables:

  • HAPROXY_TCP_LOG_FMT
  • HAPROXY_HTTP_LOG_FMT
  • HAPROXY_HTTPS_LOG_FMT

In versions prior to 2.7r1, define the environment variables yourself:

haproxy
global
setenv HAPROXY_TCP_LOG_FMT "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
setenv HAPROXY_HTTP_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
setenv HAPROXY_HTTPS_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"
haproxy
global
setenv HAPROXY_TCP_LOG_FMT "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
setenv HAPROXY_HTTP_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
setenv HAPROXY_HTTPS_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

Reference the environment variable in your log-format line. Below, we start with the HTTP log format and then append a variable named txn.myvariable to the end of the original HTTP log format:

haproxy
defaults
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.myvariable)]"
haproxy
defaults
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.myvariable)]"

A log format variable is a string prefixed by the character %. You can use them to create custom log formats. A common way to use this is to record a variable in the logs by using the var fetch method as shown below:

haproxy
frontend fe_main
bind :80
# store the 'path' in a variable named 'txn.mypath'
http-request set-var(txn.mypath) path
# logs will show the path value
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.mypath)]"
haproxy
frontend fe_main
bind :80
# store the 'path' in a variable named 'txn.mypath'
http-request set-var(txn.mypath) path
# logs will show the path value
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.mypath)]"

Avoid spaces in the variable name. A space is considered a separator. If a variable is between square brackets ([]), then it can be an expression, such as to call a fetch method to get a value. Below, we get the value from the res.cache_hit fetch method, which indicates whether the load balancer’s cache was hit:

haproxy
frontend fe_main
log-format "$HAPROXY_HTTP_LOG_FMT %[res.cache_hit]"
haproxy
frontend fe_main
log-format "$HAPROXY_HTTP_LOG_FMT %[res.cache_hit]"

HTTP response variables, those with a res. prefix, are easy because they’re certain to be available. However, variables set during the request phase are not available to the log-format directive, which executes during the response phase. To use them, you must store them in a transaction-level variable, as shown below:

haproxy
frontend fe_main
http-request set-var(txn.host) req.hdr(Host)
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.host)]"
haproxy
frontend fe_main
http-request set-var(txn.host) req.hdr(Host)
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.host)]"

You can prefix a variable with a plus or minus sign and a flag. A plus adds the flag, while a minus removes it. There are three flags available:

  • Q: adds double quotes around the value
  • X: represents the value as hexadecimal
  • E: escapes characters with a backslash, such as double quotes and backslashes

In the example below, we add double quotes around the Host header value:

haproxy
frontend fe_main
http-request set-var(txn.host) req.hdr(Host)
log-format "$HAPROXY_HTTP_LOG_FMT %{+Q}[var(txn.host)]"
haproxy
frontend fe_main
http-request set-var(txn.host) req.hdr(Host)
log-format "$HAPROXY_HTTP_LOG_FMT %{+Q}[var(txn.host)]"

Use the special variable %o to propagate its flags to all following variables in the same format string.

To emit a verbatim %, it must be preceded by another % to result in %%. HAProxy Enterprise automatically merges consecutive separators.

Log with Docker Jump to heading

When HAProxy Enterprise runs as a Docker container, you can collect logs in the following ways:

  • Forward logs to standard out, allowing you to capture them using a log aggregation tool.
  • Forward logs to a remote Syslog server, which can run in a separate container.

Log to standard out Jump to heading

This method publishes logs to standard out, allowing you to offload the collection of logs to an external tool, such as logspout.

  1. Place a log directive into the global section of your configuration file:

    • use stdout as the address
    • use format raw for a shortened log format compatible with a variety of tools
    • use local0 as the facility code
    haproxy
    global
    log stdout format raw local0
    haproxy
    global
    log stdout format raw local0
  2. Add a log global directive to a defaults section to enable the global logging rule in all subsequent listen, frontend, and backend sections:

    haproxy
    defaults
    log global
    haproxy
    defaults
    log global
  3. View logs by querying the HAProxy Enterprise container using the docker logs command:

    nix
    sudo docker logs hapee
    nix
    sudo docker logs hapee

Log to a syslog container Jump to heading

This method allows you to forward logs to a container running a Syslog server, such as Rsyslog.

  1. Place a log directive into the global section of your configuration file:

    • use the IP address or name of your Syslog container, with an optional port number
    • use local0 as the facility code
    haproxy
    global
    log rsyslog:514 local0
    haproxy
    global
    log rsyslog:514 local0
  2. Add a log global directive to a defaults section to enable the global logging rule in all subsequent listen, frontend, and backend sections:

    haproxy
    defaults
    log global
    haproxy
    defaults
    log global

    The full configuration might look like this:

    haproxy
    global
    log rsyslog:514 local0
    defaults
    log global
    frontend fe_main
    bind :80
    default_backend be_main
    backend be_main
    server web1 web1:8080 check
    haproxy
    global
    log rsyslog:514 local0
    defaults
    log global
    frontend fe_main
    bind :80
    default_backend be_main
    backend be_main
    server web1 web1:8080 check
  3. Launch containers:

    • Create the Docker network:
    nix
    sudo docker network create -d bridge my-network
    nix
    sudo docker network create -d bridge my-network
    • Run the web server container:
    nix
    sudo docker run -d --network my-network --name web1 \
    --restart unless-stopped jmalloc/echo-server
    nix
    sudo docker run -d --network my-network --name web1 \
    --restart unless-stopped jmalloc/echo-server
    • Run the Rsyslog container:
    nix
    sudo docker run -d --network my-network --name rsyslog \
    --restart unless-stopped jumanjiman/rsyslog
    nix
    sudo docker run -d --network my-network --name rsyslog \
    --restart unless-stopped jumanjiman/rsyslog
    • Run the HAProxy Enterprise container:
    nix
    sudo docker run -d --network my-network --name hapee \
    -p 80:80 -p 443:443 -p 5555:5555 --restart unless-stopped \
    -v (pwd):/etc/hapee-2.8 \
    hapee-registry.haproxy.com/haproxy-enterprise:2.8r1
    nix
    sudo docker run -d --network my-network --name hapee \
    -p 80:80 -p 443:443 -p 5555:5555 --restart unless-stopped \
    -v (pwd):/etc/hapee-2.8 \
    hapee-registry.haproxy.com/haproxy-enterprise:2.8r1
  4. View logs by querying the rsyslog container using the docker logs command:

    nix
    sudo docker logs rsyslog
    nix
    sudo docker logs rsyslog

See also Jump to heading

If this page was useful, please, Leave the feedback.