HAProxy log customization

One of HAProxy’s greatest strengths has got to be its logging system. The amount of information it provides can be invaluable when diagnosing unexpected behavior, studying your website’s traffic, or just getting a feel for how HAProxy works.

HAProxy comes with a few standard log formats that define which fields will be captured. Common wisdom says that you should add the option httplog configuration directive to your frontend or defaults section when using HAProxy as a Layer 7 proxy. Or, use option tcplog when using it as a Layer 4 proxy. Either one will give your logs an instant power-up, making them more comprehensive.

Sometimes, though, these premade settings are not exactly what you need. In that case, you can configure a custom log format. In this blog post, you’ll see how.

Find out more in our blog post “Introduction to HAProxy Logging” or by registering for our webinar: “Deep Dive Into HAProxy Logging“

Customizing the HAProxy logs

There are plenty of reasons why you might want to customize the fields captured by the HAProxy logs. For example:

• the default log format is giving you more information that you need

• you’re missing an important piece of information with the default log format

• you need to structure the fields in a way that an external tool can read them

• you rely on a standard log format and HAProxy must also comply

Luckily, it’s easy to add a template that says which fields to log and in which order. You just need to use the log-format directive. Let’s dive into its structure.

Adding a new HAProxy log format

The log-format directive goes into your defaults, frontend or listen section. It specifies a string that contains variables referring to the fields that you’d like to capture.

When you place this into your defaults section, it affects all of the proxy sections (frontend and listen) that follow. If you need a different format for a particular proxy, adding another log-format to that section will override the default.

In the next section, you’ll learn about the variables that are available.

Variables

A large part of your custom log format will likely be made up of variables that are pre-defined by HAProxy. Others you will define yourself.

In the broadest terms, all variables follows the rules below when added to a format string:

• It is preceded by a percent character: %

• It can take flags in braces {}

• If multiple flags, then they are separated by commas within the braces

• Flags may be added or removed by prefixing them with a + or a – sign

• Spaces, other than those between variables, must be escaped by a backslash

Currently, there are three flags:

• Q: quotes a string

• X: hexadecimal representation

• E: escapes characters (“, \, ]) with backslashes

Here are a few examples:

The pre-defined variables (also available in the documentation) are:

Variable	Meaning
%B	bytes read
%CC	captured request cookie
%CS	captured response cookie
%H	hostname
%HM	HTTP method (e.g. POST)
%HP	HTTP request URI without query string
%HQ	HTTP request URI query string
%HU	HTTP request URI
%HV	HTTP version (e.g. HTTP/1.1)
%ID	unique ID
%ST	status code
%T	GMT datetime
%Ta	the active time of requests (from %TR to end)
%Tc	time to establish a TCP connection to the server
%Td	Tt – (Tq + Tw + Tc + Tr)
%Tl	local datetime
%Th	connection handshake time (SSL, PROXY protocol)
%Ti	idle time before the HTTP request
%Tq	time to get the client's request
%TR	time to receive the full request from the first byte
%Tr	response time
%Ts	timestamp
%Tt	total session duration time
%U	bytes uploaded
%ac	process concurrent connections
%b	backend name
%bc	backend concurrent connections
%bi	backend source IP (HAProxy connects with)
%bp	backend source port (HAProxy connects with)
%bq	backend queue
%ci	client IP
%cp	client port
%f	frontend name
%fc	frontend concurrent connections
%fi	frontend IP
%fp	frontend port
%ft	frontend name transport (‘~’ suffix for SSL)
%lc	frontend log counter
%hr	captured request headers
%hrl	captured request headers (CLF style)
%hs	captured response headers
%hsl	captured response headers (CLF style)
%ms	accept date (milliseconds, left-padded with 0)
%pid	HAProxy process ID
%r	HTTP request
%rc	number of retries
%rt	request counter
%s	server name
%sc	server concurrent connections
%si	server IP
%sp	server port
%sq	server queue
%sslc	SSL cipher used (e.g. AES-SHA)
%sslv	SSL version (e.g. TSLv1)
%t	datetime (with millisecond resolution)
%tr	datetime of the start of HTTP request
%trg	GMT datetime of the start of HTTP request
%trl	local datetime of the start of HTTP request
%ts	termination state
%tsc	termination state with cookie status

 

The default log formats use these variables in the following ways:

TCP log format:

HTTP log format:

CLF log format:

You can create your own format by using any of the pre-defined variables in the order that you wish. The following logs the client’s IP address and port, the server’s IP address and port, the path, and the response status:

If you prefer, you can add text to give the fields more context. Placing %{+Q}o at the beginning will apply the flag, in this case, quotes, to all of the following strings.

If you prefer, you can add text to give the fields more context. Placing %{+Q}o at the beginning will apply the flag, in this case quotes, to all of the following strings.

You can also capture specific pieces of data not represented by the predefined variables. For example, you can log request headers. You’ll need to capture them first, though, and then refer to them by number. The numbers are incremented in the order in which the field was captured. Here are a few examples:

Custom variables can be logged by using the var fetch method. Note that you must prefix your variable names with txn.

Fetch methods can be included by placing them inside square brackets. In this example, the http_first_req fetch shows a 1 if this is the first request from the client, or a 0 if not.

Conclusion

In this blog post, you learned how to define a custom log format for your HAProxy logs. There are a number of pre-defined variables that you can use to build your messages. Or, you can add other data such as request headers, custom variables and explanatory text.

Want to keep up to date on similar topics? Subscribe to this blog. Also, follow us on Twitter and join the conversation on Slack!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.