Synopsis
One of the strength of HAProxy is its logging system. It is very verbose and provides many information.
HAProxy HTTP log line is briefly explained in an HAProxy Technologies memo. It’s a must have document when you have to analyze HAProxy‘s log lines to troubleshoot an issue.
An other interesting tool is HALog. It is available in HAProxy‘s sources, in the contrib directory. I’ll write later an article about it. In order to have an idea on how to use it, just have a look at HAProxy Technologies howto related to halog and HTTP analyze.
Why customizing HAProxy’s logs ???
There may be several reasons why one want to customize HAProxy’s logs:
- the default log format is too much complicated
- there are too many information in the default log format
- there is not enough information in the default log format
- third party log anaylzer can hardly understand default HAProxy log format
- logs generated by HAPorxy must be compliant to an existing format from an existing appliance in the architecture
- … add your own reason here …
That’s why, at HAProxy Technologies, we felt the need of letting our users to create their own HAProxy log-format.
As for compression in HAProxy, the job was done by Wlallemand.
HAProxy log format customization
Configuration directive
The name of the directive which allows you to generate a home made log format is simply called log-format.
Variables
The log-format directive understand variables.
A variable follows the rules below:
- it is preceded by a percent character: ‘%‘
- it can take arguments in braces ‘{}‘.
- If multiple arguments, then they are separated by commas ‘,‘ within the braces.
- Flags may be added or removed by prefixing them with a ‘+‘ or ‘–‘ sign.
- spaces ‘ ‘ must be escaped (It is considered as a separator)
Currently available flags:
- Q: quote a string
- X: hexadecimal representation (IPs, Ports, %Ts, %rt, %pid)
Currently available variables:
+---+------+-----------------------------------------------+-------------+
| R | var | field name (8.2.2 and 8.2.3 for description) | type |
+---+------+-----------------------------------------------+-------------+
| | %o | special variable, apply flags on all next var | |
+---+------+-----------------------------------------------+-------------+
| | %B | bytes_read | numeric |
| | %Ci | client_ip | IP |
| | %Cp | client_port | numeric |
| | %Bi | backend_source_ip | IP |
| | %Bp | backend_source_port | numeric |
| | %Fi | frontend_ip | IP |
| | %Fp | frontend_port | numeric |
| | %H | hostname | string |
| | %ID | unique-id | string |
| | %Si | server_IP | IP |
| | %Sp | server_port | numeric |
| | %T | gmt_date_time | date |
| | %Tc | Tc | numeric |
| H | %Tq | Tq | numeric |
| H | %Tr | Tr | numeric |
| | %Ts | timestamp | numeric |
| | %Tt | Tt | numeric |
| | %Tw | Tw | numeric |
| | %ac | actconn | numeric |
| | %b | backend_name | string |
| | %bc | beconn | numeric |
| | %bq | backend_queue | numeric |
| H | %cc | captured_request_cookie | string |
| H | %rt | http_request_counter | numeric |
| H | %cs | captured_response_cookie | string |
| | %f | frontend_name | string |
| | %ft | frontend_name_transport ('~' suffix for SSL) | string |
| | %fc | feconn | numeric |
| H | %hr | captured_request_headers default style | string |
| H | %hrl | captured_request_headers CLF style | string list |
| H | %hs | captured_response_headers default style | string |
| H | %hsl | captured_response_headers CLF style | string list |
| | %ms | accept date milliseconds | numeric |
| | %pid | PID | numeric |
| H | %r | http_request | string |
| | %rc | retries | numeric |
| | %s | server_name | string |
| | %sc | srv_conn | numeric |
| | %sq | srv_queue | numeric |
| S | %sslc| ssl_ciphers (ex: AES-SHA) | string |
| S | %sslv| ssl_version (ex: TLSv1) | string |
| H | %st | status_code | numeric |
| | %t | date_time | date |
| | %ts | termination_state | string |
| H | %tsc | termination_state with cookie status | string |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only
Log format examples
Default log format
- TCP log format
log-format %Ci:%Cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq - HTTP log format
log-format %Ci:%Cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %st %B %cc %cs %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r - CLF log format
log-format %{+Q}o %{-Q}Ci - - [%T] %r %st %B "" "" %Cp %ms %ft %b %s %Tq %Tw %Tc %Tr %Tt %tsc %ac %fc %bc %sc %rc %sq %bq %cc %cs %hrl %hsl
Home made formats
- Logging HTTP Host header, the URL, the status code, number of bytes read from server and the server response time
capture request header Host len 32 log-format %hr %r %st %B %Tr
- SSL log format with: HAProxy path (frontend, backend and server name), client information (source IP and port), SSL information (protocol version and negotiated cypher), connection termination state, including a few strings:
[sourcecode language=”text”]log-format frontend:%f %b/%s client_ip:%Ci client_port:%Cp SSL_version:%sslv SSL_cypher:%sslc %ts[/sourcecode]
I am not able to add log-format keyword in my haproxy config file.
Hi,
you must run haproxy 1.5-dev8 at least. That said, we recommend using dev17 (or latest git version)
cheers
Hi,
I want to use awstats to parse haproxy logs. So how can I get combined logs from haproxy?
Is there any example config?
And Is there any way for separating every pool’s logs in syslog-ng ? Using different facilities for every pool is not an option for me because I may use more than 25 pools
Regards
Thanks for this write up, am getting a lot of value out of my haproxy logs now!
How would one get the source port of the backend Connection? I’d like to correlate the backend Connection with the frontend Connection so we can get client IP details for non http systems that do not support the Proxy protocol:
client -> haproxy frontend -> haproxy backend -> server
84.234.2.4:23424 123.13.24.5:443 192.168.2.2:9287 192.168.10.2:443
The server only logs 192.168.2.2:9287 since it is the Proxy that makes the Connection. In ha-Proxy i only seem to be able to log the client ip and port, the frontend ip and port and the server ip and port, but not the ip and port used by the ha-Proxy to connect to the server.
Is there any way to get the 192.168.2.2:9287 ip:port information in the logs so i can correlate the logs and get the real client ip?
try to append “capture request header X-Forwarded-For len 15” into frontend block and custom log format as follows:
log-format %[capture.req.hdr(0)]:%cp\ [%t]\ %f\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ {%hrl}\ {%hsl}\ %{+Q}r
Is there a way to change the date format?
2017-11-24 vs 24-11-2017 vs 11-24-2017 vs 11/24/2017 etc
Yes you can change the date format with the ltime converter.
Haproxy logs when a particular service is down. Is there a way to log when a service is up the very first time?
When you enable logging in a backend section, HAProxy will log whenever the server operating state is changing (down or up).