One of HAProxy’s greatest strengths has got to be its logging system. The amount of information it provides can be invaluable when diagnosing unexpected behavior, studying your website’s traffic, or just getting a feel for how HAProxy works.
HAProxy comes with a few standard log formats that define which fields will be captured. Common wisdom says that you should add the option httplog
configuration directive to your frontend
or defaults
section when using HAProxy as a Layer 7 proxy. Or, use option tcplog
when using it as a Layer 4 proxy. Either one will give your logs an instant power-up, making them more comprehensive.
Sometimes, though, these premade settings are not exactly what you need. In that case, you can configure a custom log format. In this blog post, you’ll see how.
Find out more in our blog post “Introduction to HAProxy Logging” or by registering for our webinar: “Deep Dive Into HAProxy Logging“
Customizing the HAProxy Logs
There are plenty of reasons why you might want to customize the fields captured by the HAProxy logs. For example:
- the default log format is giving you more information that you need
- you’re missing an important piece of information with the default log format
- you need to structure the fields in a way that an external tool can read them
- you rely on a standard log format and HAProxy must also comply
Luckily, it’s easy to add a template that says which fields to log and in which order. You just need to use the log-format
directive. Let’s dive into its structure.
Adding a New HAProxy Log Format
The log-format
directive goes into your defaults
, frontend
or listen
section. It specifies a string that contains variables referring to the fields that you’d like to capture.
When you place this into your defaults
section, it affects all of the proxy sections (frontend
and listen
) that follow. If you need a different format for a particular proxy, adding another log-format
to that section will override the default.
In the next section, you’ll learn about the variables that are available.
Variables
A large part of your custom log format will likely be made up of variables that are pre-defined by HAProxy. Others you will define yourself.
In the broadest terms, all variables follows the rules below when added to a format string:
- It is preceded by a percent character: %
- It can take flags in braces {}
- If multiple flags, then they are separated by commas within the braces
- Flags may be added or removed by prefixing them with a + or a – sign
- Spaces, other than those between variables, must be escaped by a backslash
Currently, there are three flags:
- Q: quotes a string
- X: hexadecimal representation
- E: escapes characters (“, \, ]) with backslashes
Here are a few examples:
The pre-defined variables (also available in the documentation) are:
Variable | Meaning |
%B | bytes read |
%CC | captured request cookie |
%CS | captured response cookie |
%H | hostname |
%HM | HTTP method (e.g. POST) |
%HP | HTTP request URI without query string |
%HQ | HTTP request URI query string |
%HU | HTTP request URI |
%HV | HTTP version (e.g. HTTP/1.1) |
%ID | unique ID |
%ST | status code |
%T | GMT datetime |
%Ta | active time of requests (from %TR to end) |
%Tc | time to establish TCP connection to server |
%Td | Tt – (Tq + Tw + Tc + Tr) |
%Tl | local datetime |
%Th | connection handshake time (SSL, PROXY protocol) |
%Ti | idle time before the HTTP request |
%Tq | time to get the client request |
%TR | time to receive the full request from first byte |
%Tr | response time |
%Ts | timestamp |
%Tt | total session duration time |
%U | bytes uploaded |
%ac | process concurrent connections |
%b | backend name |
%bc | backend concurrent connections |
%bi | backend source IP (HAProxy connects with) |
%bp | backend source port (HAProxy connects with) |
%bq | backend queue |
%ci | client IP |
%cp | client port |
%f | frontend name |
%fc | frontend concurrent connections |
%fi | frontend IP |
%fp | frontend port |
%ft | frontend name transport (‘~’ suffix for SSL) |
%lc | frontend log counter |
%hr | captured request headers |
%hrl | captured request headers (CLF style) |
%hs | captured response headers |
%hsl | captured response headers (CLF style) |
%ms | accept date (milliseconds, left-padded with 0) |
%pid | HAProxy process ID |
%r | HTTP request |
%rc | number of retries |
%rt | request counter |
%s | server name |
%sc | server concurrent connections |
%si | server IP |
%sp | server port |
%sq | server queue |
%sslc | SSL cipher used (e.g. AES-SHA) |
%sslv | SSL version (e.g. TSLv1) |
%t | datetime (with millisecond resolution) |
%tr | datetime of start of HTTP request |
%trg | GMT datetime of start of HTTP request |
%trl | local datetime of start of HTTP request |
%ts | termination state |
%tsc | termination state with cookie status |
The default log formats use these variables in the following ways:
TCP log format:
HTTP log format:
CLF log format:
You can create your own format by using any of the pre-defined variables in the order that you wish. The following logs the client’s IP address and port, the server’s IP address and port, the path, and the response status:
If you prefer, you can add text to give the fields more context. Placing %{+Q}o at the beginning will apply the flag, in this case quotes, to all of the following strings.
You can also capture specific pieces of data not represented by the pre-defined variables. For example, you can log request headers. You’ll need to capture them first, though, and then refer to them by number. The numbers are incremented in the order in which the field was captured. Here are a few examples:
Custom variables can be logged by using the var
fetch method. Note that you must prefix your variable names with txn.
Fetch methods can be included by placing them inside square brackets. In this example, the http_first_req
fetch shows a 1 if this is the first request from the client, or a 0 if not.
Conclusion
In this blog post, you learned how to define a custom log format for your HAProxy logs. There are a number of pre-defined variables that you can use to build your messages. Or, you can add other data such as request headers, custom variables and explanatory text.
Want to keep up to date on similar topics? Subscribe to this blog. Also, follow us on Twitter and join the conversation on Slack! HAProxy Enterprise combines HAProxy with enterprise-class features and premium support. Contact us to learn more or sign up for a free trial today!
I am not able to add log-format keyword in my haproxy config file.
Hi,
you must run haproxy 1.5-dev8 at least. That said, we recommend using dev17 (or latest git version)
cheers
Hi,
I want to use awstats to parse haproxy logs. So how can I get combined logs from haproxy?
Is there any example config?
And Is there any way for separating every pool’s logs in syslog-ng ? Using different facilities for every pool is not an option for me because I may use more than 25 pools
Regards
Thanks for this write up, am getting a lot of value out of my haproxy logs now!
How would one get the source port of the backend Connection? I’d like to correlate the backend Connection with the frontend Connection so we can get client IP details for non http systems that do not support the Proxy protocol:
client -> haproxy frontend -> haproxy backend -> server
84.234.2.4:23424 123.13.24.5:443 192.168.2.2:9287 192.168.10.2:443
The server only logs 192.168.2.2:9287 since it is the Proxy that makes the Connection. In ha-Proxy i only seem to be able to log the client ip and port, the frontend ip and port and the server ip and port, but not the ip and port used by the ha-Proxy to connect to the server.
Is there any way to get the 192.168.2.2:9287 ip:port information in the logs so i can correlate the logs and get the real client ip?
try to append “capture request header X-Forwarded-For len 15” into frontend block and custom log format as follows:
log-format %[capture.req.hdr(0)]:%cp\ [%t]\ %f\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ {%hrl}\ {%hsl}\ %{+Q}r
Is there a way to change the date format?
2017-11-24 vs 24-11-2017 vs 11-24-2017 vs 11/24/2017 etc
Yes you can change the date format with the ltime converter.
Haproxy logs when a particular service is down. Is there a way to log when a service is up the very first time?
When you enable logging in a backend section, HAProxy will log whenever the server operating state is changing (down or up).