Manage HAProxy Enterprise logs

HAProxy Enterprise generates two types of logs: access logs and administrative logs.

Enable access logs Jump to heading #

Access logs, also called traffic logs, record information about client connections and requests. The logs are stored in the /var/log/hapee-<VERSION> directory and have file names prefixed with lb-access- plus a timestamp.

To enable access logs:

1. Add the following log directive in the global section of the HAProxy Enterprise configuration file, if one does not already exist:

   haproxy
   global
     log 127.0.0.1 local0 

   haproxy
   global
     log 127.0.0.1 local0 

   This definition, which does not set a severity level for events, captures all events and tags them with the local0 syslog facility code. An rsyslog configuration file at /etc/rsyslog.d/hapee-<VERSION>-lb.conf routes these events to the access log file.

   You can change the IP address to send logs to a remote rsyslog server over UDP.

2. In your frontend or defaults section, add a log global directive, which applies the log directive from the global section to the frontend.

   haproxy
   frontend fe_main
     log global

   haproxy
   frontend fe_main
     log global

Enable administrative logs Jump to heading #

Administrative logs record information about load balancer process events such as when the load balancer starts or stops. The logs are stored in the /var/log/hapee-<VERSION> directory and have file names prefixed with lb-admin- plus a timestamp.

To enable administrative logs:

1. Add the following log directive in the global section of the HAProxy Enterprise configuration file, if one does not already exist:

   haproxy
   global
     log 127.0.0.1 local1 notice

   haproxy
   global
     log 127.0.0.1 local1 notice

   This definition, which sets its severity level to notice, captures most log events but omits the more verbose traffic events captured by the access logs. It tags those events with the syslog facility code local1. An rsyslog configuration file at /etc/rsyslog.d/hapee-<VERSION>-lb.conf routes these events to the admin log file. Basically, they have the same contents as access logs but without the info level details about connection and request events.

   You can change the IP address to send logs to a remote rsyslog server over UDP.

2. In your frontend or defaults section, add a log global directive, which applies the log directive from the global section to the frontend.

   haproxy
   frontend fe_main
     log global

   haproxy
   frontend fe_main
     log global

Enable log compression and rotation Jump to heading #

The hasavelog utility, disabled by default, can automatically compress and delete old HAProxy Enterprise logs. The enabled, out-of-the-box defaults for the hasavelog utility will:

• compress log files using the gzip compression method
• operate in HAProxy Enterprise mode (enforces nocycle, nodate, notouch, nocreate, and noclean)
• create and write a new compressed log file daily into the directory /var/log/hapee-<VERSION>
• operate in quiet mode
• keep all log files indefinitely unless specified

To enable the hasavelog utility:

1. Open the hasavelog utility’s configuration file. Located at:

   Debian/Ubuntu RHEL

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

2. The HASAVELOG_ENABLE field accepts either 1 (enabled) or 0 (disabled). Set HASAVELOG_ENABLE="1" and save.

3. Start the hasavelog service with:

   nix
   sudo systemctl start hapee-<VERSION>-hasavelog

   nix
   sudo systemctl start hapee-<VERSION>-hasavelog

Tune the hasavelog utility with additional arguments Jump to heading #

The HASAVELOG_ARGS field tunes the hasavelog utility. HASAVELOG_ARGS has a default configuration of -e -r /var/log/hapee-<VERSION> -q, where logs are stored in /var/log/hapee-<VERSION>.

HASAVELOG_ARGS can set the following arguments:

To tune HASAVELOG_ARGS:

1. Open the hasavelog utility’s configuration file. Located at:

   Debian/Ubuntu RHEL

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

2. Change the HASAVELOG_ARGS field to include the arguments you want to set and save. For example:

   text
   HASAVELOG_ARGS="-e -r /var/log/hapee-@@HAPEE_MAJORVERSION@@ -q -Z -s 1"

   text
   HASAVELOG_ARGS="-e -r /var/log/hapee-@@HAPEE_MAJORVERSION@@ -q -Z -s 1"

   In addition to the out-of-the-box defaults, compresses log files over 1 MB in size with the pigz compression method.

3. Restart the hasavelog service after any saved changes to the hasavelog utility configuration file:

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

Set the retention period for logs Jump to heading #

The HASAVELOG_RETAIN field sets the retention period. It keeps all log files indefinitely with the default value of an empty string (""). Using an integer, set HASAVELOG_RETAIN to delete logs older than that number of days.

1. Open the hasavelog utility’s configuration file. Located at:

   Debian/Ubuntu RHEL

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

2. Change the value of HASAVELOG_RETAIN to the number of days you would like to keep log files for. For example, setting HASAVELOG_RETAIN="30" will automatically continue to delete any logs older than 30 days.

3. Restart the hasavelog service after any saved changes to the hasavelog utility configuration file:

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

Enable audit logging Jump to heading #

The HASAVELOG_AUDIT field, disabled by default, allows the hasavelog utility to log file deletions for compliance. It’s printed to stdout. The HASAVELOG_AUDIT field accepts either 1 (enabled) or 0 (disabled).

When the field is enabled, the hasavelog utility offers more verbose logging:

1. Open the hasavelog utility’s configuration file. Located at:

   Debian/Ubuntu RHEL

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/default/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

   nix
   /etc/sysconfig/hapee-<VERSION>-hasavelog

2. The HASAVELOG_AUDIT field accepts either 1 (enabled) or 0 (disabled). Set HASAVELOG_AUDIT="1" and save.

3. Restart the hasavelog service after any changes to the hasavelog utility configuration file:

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

   nix
   sudo systemctl restart hapee-<VERSION>-hasavelog

An example output with HASAVELOG_AUDIT="1" and HASAVELOG_RETAIN="2":

nix
systemctl status hapee-<VERSION>-hasavelog.service

nix
systemctl status hapee-<VERSION>-hasavelog.service

output
text
[...]
Jul 22 14:34:27 hostname1 hasavelog[1569]: hasavelog: `find -delete -print' in /var/log/hapee-<VERSION>
Jul 22 14:34:27 hostname1 hasavelog[1575]: hasavelog: /var/log/hapee-<VERSION>/lb-access-20240719.log
Jul 22 14:34:27 hostname1 hasavelog[1575]: hasavelog: /var/log/hapee-<VERSION>/lb-admin-20240719.log
Jul 22 14:34:27 hostname1 systemd[1]: hapee-<VERSION>-hasavelog.service: Succeeded.
Jul 22 14:34:27 hostname1 systemd[1]: Finished Compress HAProxy Enterprise log files.

output
text
[...]
Jul 22 14:34:27 hostname1 hasavelog[1569]: hasavelog: `find -delete -print' in /var/log/hapee-<VERSION>
Jul 22 14:34:27 hostname1 hasavelog[1575]: hasavelog: /var/log/hapee-<VERSION>/lb-access-20240719.log
Jul 22 14:34:27 hostname1 hasavelog[1575]: hasavelog: /var/log/hapee-<VERSION>/lb-admin-20240719.log
Jul 22 14:34:27 hostname1 systemd[1]: hapee-<VERSION>-hasavelog.service: Succeeded.
Jul 22 14:34:27 hostname1 systemd[1]: Finished Compress HAProxy Enterprise log files.

The highlighted lines represent what you will see with audit loggging enabled. On July 22nd, the hasavelog utility deleted the logs from July 19th because it is over 2 days old. It then printed the deletion events to stdout.

Set logging levels Jump to heading #

Log levels are the same as the severity levels used by the syslog service. Below, we set the level to notice, which captures all events at that level and above (up to emerg):

haproxy
global
  log 127.0.0.1 local1 notice

haproxy
global
  log 127.0.0.1 local1 notice

HAProxy Enterprise tags each loggable event with a severity level. For example, it categorizes log messages related to connections and HTTP requests with the info severity level. Other events are categorized using one of the other, less verbose levels. From most to least verbose, the severity levels are:

Set conditional log levels Jump to heading #

Sometimes, you’ll want to capture a more verbose log entry for a particular request or response, such as when the server returned an error. To change the log level for individual requests or responses that match specified criteria, use one of these directives, depending on whether you’re targeting the request phase or response phase, and whether your’re in HTTP or TCP mode:

• http-request set-log-level <level> [ { if | unless } <condition> ]
• http-response set-log-level <level> [ { if | unless } <condition> ]
• tcp-request content set-log-level <level> [ { if | unless } <condition> ]
• tcp-response content set-log-level <level> [ { if | unless } <condition> ]

Using these directives, you can set the log level to any of the eight syslog levels or to the special level silent, which disables logging for this request/response. This rule is not final so the last matching rule wins.

Specify a condition using the ACL syntax:

haproxy
frontend www
  bind :80
  acl failed_request status 400 401 403 404 405 408 429 500 503
  http-response set-log-level err if failed_request

haproxy
frontend www
  bind :80
  acl failed_request status 400 401 403 404 405 408 429 500 503
  http-response set-log-level err if failed_request

Log ASAP Jump to heading #

By default, logs are emitted when the session terminates. That is, when all log values are finalized. Capturing some log values, such as total transfer time and message size, can delay generation of the log content, particularly for a large file transfer.

To avoid such delays, you can choose to log available values as quickly as possible by specifying the logasap option. With this option, the load balancer creates the log message as soon as the server connection has been established in mode tcp or as soon as the server sends the complete headers in mode http.

Values not finalized at this time are total time and total size metrics, and they are prefixed with a plus sign “+” in the log to indicate that they are not true, final values.

Enable early logging by adding option logasap to the defaults or frontend section, as shown below. In this example, we also capture the Content-Length header in the logs so that the logs at least indicate how many bytes are expected to be transferred.

hapee-lb.cfg
haproxy
frontend fe_main
  bind 0.0.0.0:80
  mode http
  log global
  option httplog
  option logasap
  http-request capture req.hdr(Content-Length) len 15

hapee-lb.cfg
haproxy
frontend fe_main
  bind 0.0.0.0:80
  mode http
  log global
  option httplog
  option logasap
  http-request capture req.hdr(Content-Length) len 15

output
text
>>> Feb  6 12:14:14 localhost \
      hapee-lb[14389]: 10.0.1.2:33317 [06/Feb/2019:12:14:14.655] fe_main \
      static/srv1 9/10/7/14/+30 200 +243 - - ---- 3/1/1/1/0 1/0 \
      "GET /image.iso HTTP/1.0"

output
text
>>> Feb  6 12:14:14 localhost \
      hapee-lb[14389]: 10.0.1.2:33317 [06/Feb/2019:12:14:14.655] fe_main \
      static/srv1 9/10/7/14/+30 200 +243 - - ---- 3/1/1/1/0 1/0 \
      "GET /image.iso HTTP/1.0"

Configure conditional logging Jump to heading #

Using the when() converter, you can configure the load balancer to emit certain log messages only when certain conditions are met, for example, on error or because of another abnormal state. See when() converter for more information.

Log profiles Jump to heading #

In addition to defining log formats individually throughout a load balancer configuration, you can define them in named log profiles that you can then apply wherever needed.

You can define as many log profiles as you want, giving each a name allowing you to apply it independently. With log profiles, you can choose the one best suited for each log server and even emit logs to multiple servers at the same time, each with its own format. Log profiles also allow you to write log messages when different events occur, such as when accepting a connection, receiving a request, connecting to a backend server, and receiving a response.

Log profiles present plenty of opportunities:

• Create a log profile for emitting timing information to see how long HAProxy Enterprise took to handle a request.
• Create a log profile that emits the maximum amount of data for use in debugging.
• Switch the log format just by changing the profile argument on the log line.
• Reuse profiles across multiple frontends.
• Decide whether you want to emit messages for every step defined in a profile or for only some of them by setting the log-steps directive.

In the example below, we define two log profiles: one named syslog, which uses the syslog format, and another named json, which uses JSON. For syslog, we set the log-tag directive inside to change the syslog header’s tag field, to give a hint to the syslog server about how to process the message. Notice that we also get to choose when to emit the log message. We’re emitting the log message on the close event, when HAProxy has finalized the request-response transaction and has access to all of the data:

hapee-lb.cfg
haproxy
log-profile syslog
  log-tag "haproxy"
  on close format "$HAPROXY_HTTP_LOG_FMT"
log-profile json
  on close format "%{+json}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"

hapee-lb.cfg
haproxy
log-profile syslog
  log-tag "haproxy"
  on close format "$HAPROXY_HTTP_LOG_FMT"
log-profile json
  on close format "%{+json}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"

Our frontend uses both log profiles. By setting the profile argument on each log line, the frontend will send syslog to one log server and json to another. It’s required to also enable logging in the frontend, which we accomplish by setting option httplog, but option tcplog would also work.

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  option httplog
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info
  log 10.0.0.11:9200 format raw     profile json   local0 info

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  option httplog
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info
  log 10.0.0.11:9200 format raw     profile json   local0 info

By default, HAProxy emits a log message when the close event fires, but you can emit messages on other events, too. The supported events are accept, close, connect, error, request, and response.

By tweaking the syslog profile to include more on lines, we have logged a message at each event in HAProxy’s processing:

hapee-lb.cfg
haproxy
log-profile syslog
  on accept format "Client connected. IP: %ci"
  on request format "Request received. %r"
  on connect format "Connected to backend server. %b / %s"
  on response format "Response received."
  on close format "$HAPROXY_HTTP_LOG_FMT"
  on error format "Error!"

hapee-lb.cfg
haproxy
log-profile syslog
  on accept format "Client connected. IP: %ci"
  on request format "Request received. %r"
  on connect format "Connected to backend server. %b / %s"
  on response format "Response received."
  on close format "$HAPROXY_HTTP_LOG_FMT"
  on error format "Error!"

To enable these extra messages, set the log-steps directive to all or to a comma-separated list of events:

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  log-steps all
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  log-steps all
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info

Custom logs any time Jump to heading #

The do-log action allows you to emit custom log messages at any point in the processing of a request or response. Add the do-log action where desired in your load balancer configuration. In the example below, we set a variable named req.log_msg and then use a do-log action to log the variable value.

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info
  acl is_evil_client req.hdr(user-agent) -m sub evil
  http-request set-var(req.log_msg) str("Blocking evil client") if is_evil_client
  http-request do-log
  http-request deny if is_evil_client

hapee-lb.cfg
haproxy
frontend mysite
  bind :80
  log 10.0.0.10:514  format rfc5424 profile syslog local0 info
  acl is_evil_client req.hdr(user-agent) -m sub evil
  http-request set-var(req.log_msg) str("Blocking evil client") if is_evil_client
  http-request do-log
  http-request deny if is_evil_client

Update your syslog log-profile section (see Log profiles) so that it includes the line on http-req, which defines the log format to use whenever http-request do-log is called.

The following example log format prints the value of the variable req.log_msg:

hapee-lb.cfg
haproxy
log-profile syslog
  log-tag "haproxy"
  on close format "$HAPROXY_HTTP_LOG_FMT"
  on http-req format "%[var(req.log_msg)]"

hapee-lb.cfg
haproxy
log-profile syslog
  log-tag "haproxy"
  on close format "$HAPROXY_HTTP_LOG_FMT"
  on http-req format "%[var(req.log_msg)]"

Your log will show the custom log message:

text
Blocking evil client

text
Blocking evil client

The do-log action works with other directives, too. Each matches up with a step in the log-profile section:

Set the log format Jump to heading #

By default, HAProxy Enterprise logs are sparse, capturing minimal information about each TCP connection in the access logs:

text
>>> Jan 27 21:15:06 localhost hapee-lb[2901]: Connect from 192.168.50.1:61459 to 192.168.50.25:80 \
      (fe_main/HTTP)

text
>>> Jan 27 21:15:06 localhost hapee-lb[2901]: Connect from 192.168.50.1:61459 to 192.168.50.25:80 \
      (fe_main/HTTP)

The information includes:

To get a more detailed log, use one of the directives described in the following sections.

TCP log format Jump to heading #

The option tcplog directive is suitable for frontend sections that either specify mode tcp or don’t set mode at all.

haproxy
defaults
  mode tcp
  option tcplog

haproxy
defaults
  mode tcp
  option tcplog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Jan 27 21:22:53 localhost hapee-lb[3042]: 192.168.50.1:61535 [27/Jan/2021:21:22:10.944] \
      fe_main be_servers/s1 1/0/42404 2895 cD 1/1/0/0/0 0/0

output
text
>>> Jan 27 21:22:53 localhost hapee-lb[3042]: 192.168.50.1:61535 [27/Jan/2021:21:22:10.944] \
      fe_main be_servers/s1 1/0/42404 2895 cD 1/1/0/0/0 0/0

The information included is detailed in the TCP log format section of the reference guide.

The tcplog log format is equivalent to the following log-format definition, if you were to set it yourself:

haproxy
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

haproxy
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

HTTP log format Jump to heading #

The option httplog directive is suitable for frontend sections that specify mode http. This option provides the same information as the tcplog option, but with additional HTTP details.

haproxy
defaults
  mode http
  option httplog

haproxy
defaults
  mode http
  option httplog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Jan 27 21:52:56 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
      fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} "GET / HTTP/1.1"

output
text
>>> Jan 27 21:52:56 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
      fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} "GET / HTTP/1.1"

The information included is detailed in the HTTP log format section of the reference guide.

The httplog log format is equivalent to the following log-format definition, if you were to set it yourself:

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

CLF log format for TCP Jump to heading #

If you intend to pass the log to a log analyzer that supports the CLF format, you can specify the optional clf argument:

haproxy
defaults
  mode tcp
  option tcplog clf

haproxy
defaults
  mode tcp
  option tcplog clf

The CLF format provides the same details as the tcplog format, but in the order required for CLF parsing.

For example, option tcplog clf is equivalent to the following log-format definition:

haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc  %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"

haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc  %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"

The CLF format is intended for HTTP data and includes fields not found in TCP data. These fields are set to static values in the log. The HTTP request will show as TCP and the response code will show as 000.

By default, the CLF format reports timestamps with whole-second granularity. To report timestamps with millisecond granularity instead, use the option tcplog directive (without appending clf), or use a custom log format that reports %tr.

CLF log format for HTTP Jump to heading #

If you intend to pass the log to a log analyzer that supports the CLF format, you can specify the optional clf argument:

haproxy
defaults
  mode http
  option httplog clf

haproxy
defaults
  mode http
  option httplog clf

The CLF format provides the same details as the httplog format, but in the order required for CLF parsing.

For example, option httplog clf is equivalent to the following log-format definition:

haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc  %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"

haproxy
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc  %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"

By default, the CLF format reports timestamps with whole-second granularity. To report timestamps with millisecond granularity instead, use the option httplog directive (without appending clf), or use a custom log format that reports %tr.

HTTPS log format Jump to heading #

The option httpslog directive is suitable for HTTP over TLS connections. This option provides the same information as the option httplog directive, but with additional TLS details.

haproxy
defaults
  mode http
  option httpslog

haproxy
defaults
  mode http
  option httpslog

After reloading your configuration, you’ll see the following log format:

output
text
>>> Feb  6 12:14:14 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
      fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} \
      "GET / HTTP/1.1" 0/0/0/0/0 1wt.eu/TLSv1.3/TLS_AES_256_GCM_SHA384

output
text
>>> Feb  6 12:14:14 localhost hapee-lb[3098]: 192.168.50.1:61818 [27/Jan/2021:21:52:56.086] \
      fe_main be_servers/s1 0/0/1/1/2 200 517 - - ---- 1/1/0/0/0 0/0 {1wt.eu} {} \
      "GET / HTTP/1.1" 0/0/0/0/0 1wt.eu/TLSv1.3/TLS_AES_256_GCM_SHA384

The information included is detailed in the HTTPS log format section of the reference guide.

The httpslog log format is equivalent to the following log-format definition:

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

haproxy
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

Custom log format Jump to heading #

You can define your own log format and record a custom set of information about connections or HTTP requests. Your log format can include variables and values from fetch methods.

Use the log-format directive to create a new log format in your defaults or frontend section. It specifies a string that contains variables referring to the fields that you’d like to capture.

Often, you will want to keep the existing information collected by the premade log format rules and append new information to them. From version 2.7r1 onward, to make that easier, you can use one of the following environment variables:

• HAPROXY_TCP_LOG_FMT
• HAPROXY_HTTP_LOG_FMT
• HAPROXY_HTTPS_LOG_FMT

In versions prior to 2.7r1, define the environment variables yourself:

haproxy
global
  setenv HAPROXY_TCP_LOG_FMT "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
  setenv HAPROXY_HTTP_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
  setenv HAPROXY_HTTPS_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

haproxy
global
  setenv HAPROXY_TCP_LOG_FMT "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
  setenv HAPROXY_HTTP_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
  setenv HAPROXY_HTTPS_LOG_FMT "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"

Reference the environment variable in your log-format line. Below, we start with the HTTP log format and then append a variable named txn.myvariable to the end of the original HTTP log format:

haproxy
defaults
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.myvariable)]"

haproxy
defaults
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.myvariable)]"

A log format variable is a string prefixed by the character %. You can use them to create custom log formats. A common way to use this is to record a variable in the logs by using the var fetch method as shown below:

haproxy
frontend fe_main
  bind :80
  
  # store the 'path' in a variable named 'txn.mypath'
  http-request set-var(txn.mypath) path
  
  # logs will show the path value
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.mypath)]"

haproxy
frontend fe_main
  bind :80
  
  # store the 'path' in a variable named 'txn.mypath'
  http-request set-var(txn.mypath) path
  
  # logs will show the path value
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.mypath)]"

Avoid spaces in the variable name. A space is considered a separator. If a variable is between square brackets ([ … ]), then it can be an expression, such as to call a fetch method to get a value. Below, we get the value from the res.cache_hit fetch method, which indicates whether the load balancer’s cache was hit:

haproxy
frontend fe_main
  log-format "$HAPROXY_HTTP_LOG_FMT %[res.cache_hit]"

haproxy
frontend fe_main
  log-format "$HAPROXY_HTTP_LOG_FMT %[res.cache_hit]"

HTTP response variables, those with a res. prefix, are easy because they’re certain to be available. However, variables set during the request phase are not available to the log-format directive, which executes during the response phase. To use them, you must store them in a transaction-level variable, as shown below:

haproxy
frontend fe_main
  http-request set-var(txn.host) req.hdr(Host)
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.host)]"

haproxy
frontend fe_main
  http-request set-var(txn.host) req.hdr(Host)
  log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.host)]"

You can prefix a variable with a plus or minus sign and a flag. A plus adds the flag, while a minus removes it. There are three flags available:

• Q: adds double quotes around the value
• X: represents the value as hexadecimal
• E: escapes characters with a backslash, such as double quotes and backslashes

In the example below, we add double quotes around the Host header value:

haproxy
frontend fe_main
  http-request set-var(txn.host) req.hdr(Host)
  log-format "$HAPROXY_HTTP_LOG_FMT %{+Q}[var(txn.host)]"

haproxy
frontend fe_main
  http-request set-var(txn.host) req.hdr(Host)
  log-format "$HAPROXY_HTTP_LOG_FMT %{+Q}[var(txn.host)]"

Use the special variable %o to propagate its flags to all following variables in the same format string.

To emit a verbatim %, it must be preceded by another % to result in %%. HAProxy Enterprise automatically merges consecutive separators.

Log to JSON Jump to heading #

To define a custom log format that returns a JSON-formatted string:

• Prefix your log-format string with %{+json}o.
• Prefix each field with a key surrounded by parentheses.

Example:

haproxy
frontend mysite
  bind :80
  log-format "%{+json}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"
  use_backend webservers

haproxy
frontend mysite
  bind :80
  log-format "%{+json}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"
  use_backend webservers

Entries in the log will look like this:

json
{"client_ip": "172.21.0.1", "client_port": 57526, "request_date": "05/Jun/2024:20:59:57.698", "frontend_name": "mysite", "backend_name": "webservers", "server_name": "web1", "time_to_receive": 0, "time_waiting": 0, "time_to_connect": 0, "time_server_response": 0, "time_active": 0, "status_code": 200, "bytes_read": 993, "request_cookies": "", "response_cookies": "", "termination_state": "----", "process_active_connections": 1, "frontend_active_connections": 1, "backend_active_connections": 0, "server_active_connections": 0, "retries": 0, "server_queue": 0, "backend_queue": 0, "request_headers": null, "response_headers": null, "request_line": "GET / HTTP/1.1"}

json
{"client_ip": "172.21.0.1", "client_port": 57526, "request_date": "05/Jun/2024:20:59:57.698", "frontend_name": "mysite", "backend_name": "webservers", "server_name": "web1", "time_to_receive": 0, "time_waiting": 0, "time_to_connect": 0, "time_server_response": 0, "time_active": 0, "status_code": 200, "bytes_read": 993, "request_cookies": "", "response_cookies": "", "termination_state": "----", "process_active_connections": 1, "frontend_active_connections": 1, "backend_active_connections": 0, "server_active_connections": 0, "retries": 0, "server_queue": 0, "backend_queue": 0, "request_headers": null, "response_headers": null, "request_line": "GET / HTTP/1.1"}

Log to CBOR Jump to heading #

To define a custom log format that returns a CBOR-formatted string:

• Prefix your log-format string with %{+cbor}o.
• Prefix each field with a key surrounded by parentheses.

Example:

haproxy
frontend mysite
  bind :80
  log-format "%{+cbor}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"
  use_backend webservers

haproxy
frontend mysite
  bind :80
  log-format "%{+cbor}o %(client_ip)ci %(client_port)cp %(request_date)tr %(frontend_name)ft %(backend_name)b %(server_name)s %(time_to_receive)TR %(time_waiting)Tw %(time_to_connect)Tc %(time_server_response)Tr %(time_active)Ta %(status_code)ST %(bytes_read)B %(request_cookies)CC %(response_cookies)CS %(termination_state)tsc %(process_active_connections)ac %(frontend_active_connections)fc %(backend_active_connections)bc %(server_active_connections)sc %(retries)rc %(server_queue)sq %(backend_queue)bq %(request_headers)hr %(response_headers)hs %(request_line)r"
  use_backend webservers

Entries in the log will look like this (line breaks added for clarity):

text
BF69636C69656E745F69706A3137322E32312E302E316B636C69656E745F706F727419E030
6C726571756573745F64617465781830352F4A756E2F323032343A32313A30323A33392E33
34346D66726F6E74656E645F6E616D657F666D7973697465FF6C6261636B656E645F6E616D
656A776562736572766572736B7365727665725F6E616D6564776562316F74696D655F746F
5F72656365697665006C74696D655F77616974696E67006F74696D655F746F5F636F6E6E65
6374007474696D655F7365727665725F726573706F6E7365006B74696D655F616374697665
006B7374617475735F636F646518C86A62797465735F726561641903E16F72657175657374
5F636F6F6B696573F670726573706F6E73655F636F6F6B696573F6717465726D696E617469
6F6E5F7374617465642D2D2D2D781A70726F636573735F6163746976655F636F6E6E656374
696F6E7301781B66726F6E74656E645F6163746976655F636F6E6E656374696F6E7301781A
6261636B656E645F6163746976655F636F6E6E656374696F6E730078197365727665725F61
63746976655F636F6E6E656374696F6E73006772657472696573006C7365727665725F7175
657565006D6261636B656E645F7175657565006F726571756573745F68656164657273F670
726573706F6E73655F68656164657273F66C726571756573745F6C696E657

text
BF69636C69656E745F69706A3137322E32312E302E316B636C69656E745F706F727419E030
6C726571756573745F64617465781830352F4A756E2F323032343A32313A30323A33392E33
34346D66726F6E74656E645F6E616D657F666D7973697465FF6C6261636B656E645F6E616D
656A776562736572766572736B7365727665725F6E616D6564776562316F74696D655F746F
5F72656365697665006C74696D655F77616974696E67006F74696D655F746F5F636F6E6E65
6374007474696D655F7365727665725F726573706F6E7365006B74696D655F616374697665
006B7374617475735F636F646518C86A62797465735F726561641903E16F72657175657374
5F636F6F6B696573F670726573706F6E73655F636F6F6B696573F6717465726D696E617469
6F6E5F7374617465642D2D2D2D781A70726F636573735F6163746976655F636F6E6E656374
696F6E7301781B66726F6E74656E645F6163746976655F636F6E6E656374696F6E7301781A
6261636B656E645F6163746976655F636F6E6E656374696F6E730078197365727665725F61
63746976655F636F6E6E656374696F6E73006772657472696573006C7365727665725F7175
657565006D6261636B656E645F7175657565006F726571756573745F68656164657273F670
726573706F6E73655F68656164657273F66C726571756573745F6C696E657

Log with Docker Jump to heading #

When HAProxy Enterprise runs as a Docker container, you can collect logs in the following ways:

• Forward logs to standard out, allowing you to capture them using a log aggregation tool.
• Forward logs to a remote Syslog server, which can run in a separate container.

Log to standard out Jump to heading #

This method publishes logs to standard out, allowing you to offload the collection of logs to an external tool, such as logspout.

1. Place a log directive into the global section of your configuration file:

   • use stdout as the address
   • use format raw for a shortened log format compatible with a variety of tools
   • use local0 as the facility code

   haproxy
   global
     log stdout format raw local0

   haproxy
   global
     log stdout format raw local0

2. Add a log global directive to a defaults section to enable the global logging rule in all subsequent listen, frontend, and backend sections:

   haproxy
   defaults
     log global

   haproxy
   defaults
     log global

3. View logs by querying the HAProxy Enterprise container using the docker logs command:

   nix
   sudo docker logs hapee

   nix
   sudo docker logs hapee

Log to a syslog container Jump to heading #

This method allows you to forward logs to a container running a Syslog server, such as Rsyslog.

1. Place a log directive into the global section of your configuration file:

   • use the IP address or name of your Syslog container, with an optional port number
   • use local0 as the facility code

   haproxy
   global
     log rsyslog:514 local0

   haproxy
   global
     log rsyslog:514 local0

2. Add a log global directive to a defaults section to enable the global logging rule in all subsequent listen, frontend, and backend sections:

   haproxy
   defaults
     log global

   haproxy
   defaults
     log global

   The full configuration might look like this:

   haproxy
   global
     log rsyslog:514 local0
   defaults
     log global
   frontend fe_main
     bind :80
     default_backend be_main
   backend be_main
     server web1 web1:8080 check

   haproxy
   global
     log rsyslog:514 local0
   defaults
     log global
   frontend fe_main
     bind :80
     default_backend be_main
   backend be_main
     server web1 web1:8080 check

3. Launch containers:

   • Create the Docker network:

   nix
   sudo docker network create -d bridge my-network

   nix
   sudo docker network create -d bridge my-network

   • Run the web server container:

   nix
   sudo docker run -d --network my-network --name web1 \
     --restart unless-stopped jmalloc/echo-server

   nix
   sudo docker run -d --network my-network --name web1 \
     --restart unless-stopped jmalloc/echo-server

   • Run the Rsyslog container:

   nix
   sudo docker run -d --network my-network --name rsyslog \
     --restart unless-stopped jumanjiman/rsyslog

   nix
   sudo docker run -d --network my-network --name rsyslog \
     --restart unless-stopped jumanjiman/rsyslog

   • Run the HAProxy Enterprise container:

   nix
   sudo docker run -d --network my-network --name hapee \
     -p 80:80 -p 443:443 -p 5555:5555 --restart unless-stopped \
     -v (pwd):/etc/hapee-3.1 \
     hapee-registry.haproxy.com/haproxy-enterprise:3.1r1

   nix
   sudo docker run -d --network my-network --name hapee \
     -p 80:80 -p 443:443 -p 5555:5555 --restart unless-stopped \
     -v (pwd):/etc/hapee-3.1 \
     hapee-registry.haproxy.com/haproxy-enterprise:3.1r1

4. View logs by querying the rsyslog container using the docker logs command:

   nix
   sudo docker logs rsyslog

   nix
   sudo docker logs rsyslog

See also Jump to heading #

• For complete information about logging configuration, see Logging reference.