Performing Health Checks

Understanding the Server's Operational State
Configuring a Health Check
- Set the IP address, port, and protocol
- Defining health check parameters
Check a TCP Port
Check an SSL Port
Check an HTTP service
Check an LDAP service
Check a MySql Service
Check a PgSQL Service
Check a Redis Service
Check an SMTP service
Check any service

Before HAProxy forwards traffic to servers, we recommend that you configure it to perform health checks on the service hosted by each server in a farm.

Note

It is important to set up health checks as close as possible to the load-balanced application.

Understanding the Server's Operational State

From an HAProxy point of view, a server can have different operational states.

A server state is defined by the latest check results. The operational states are:

UP	Server is operational
UP - transitionally DOWN	Server is currently considered as operational, but the last check failed. Hence, the server is transitioning to the DOWN state
DOWN - transitionally UP	Server is currently considered as non-operational, but the last check succeeded. Hence, the server is transitioning to the UP state
DOWN	Server is non-operational

The operational state changes are triggered by health check parameters.

The diagram above shows the different server states and how HAProxy parameters interact with them.

Configuring a Health Check

The minimum configuration for a health check is the check keyword on a server line. In order to run, a health check requires at least an IP address and a TCP port from the server.

Set the IP address, port, and protocol

In HAProxy, all servers configured in a backend get the same health check and must return the same type of response.

A check needs at least the following information:

Information	Default value	Alternative value
IP address	Server IP address	Address provided by server's `addr` parameter
TCP port	Server port	TCP port provided by server's `port` parameter
Proxy protocol	Server `send-proxy` parameter	Follow flag `check-send-proxy` from the `server` line
Ciphered	connection Server's `ssl` parameter	Follow flag `check-ssl` from the `server` line
Protocol	TCP handshake	SSL, HTTP, LDAP, MySql, PgSQL, redis, SMTP, generic Send/Expect
Valid response	Depends on the protocol used

Note

When set, addr and port parameters have precedence over the server's IP and port.

Defining health check parameters

Check Interval

All the keywords below apply to the server or default-server directives which you can use to set up a health check frequency based on the server's state:

Parameter	Description
`inter`	Sets the interval between two consecutive health checks. If not specified, the default value is 2s.
`fastinter`	Sets the interval between two consecutive health checks when the server is in any of the transition states: UP - transitionally DOWN or DOWN - transitionally UP. If not set, then inter is used.
`downinter`	Sets the interval between two consecutive health checks when the server is in the DOWN state. If not set, then inter is used.

Initial check (startup)

All the keywords below apply to the global section and can be used to set up HAProxy's behavior when running the first check after startup:

Keyword	Description
`max-spread-checks <delay>`	When starting up, HAProxy administers the first health checks for a farm over the `inter` period. When `inter` is very long, some servers may appear UP after a very long time. This parameter allows the reduction of this initial period to `<delay>`.
`spread-checks <0..50, in percent>`	Adds some randomness on the interval delay between two consecutive checks to avoid sending health checks at intervals that are too regular. If not set, the default value is 0.

Check buffer size

Some health checks may need to look for data in the response body. By default, HAProxy reads only the first 16384 bytes of the response and ignores the rest. If you want to search for information above this value, you must set up the tune.chksize parameter in the global section.

Response validation

All the keywords below apply to the server or default-server directives and can be used to tell HAProxy to consider the number of positive or negative valid checks before it changes a server's state:

Keyword	Description
`rise <count>`	Number of consecutive valid health checks before considering the server as UP. Default value is 2
`fall`	Number of consecutive invalid health checks before considering the server as DOWN. Default value is 3

Check timeouts

All the keywords below apply to the backend or defaults sections and can be used to tell HAProxy how long to wait for a server response:

Keyword	Description
`timeout check <timeout>`	Time allowed for the server to answer the check. If both `inter` and `timeout check` are set, then the smaller value of the two is used after the TCP connection is established When a `timeout check` is not set, `inter` starts with the establishment of the TCP connection.

Keyword

Description

timeout check <timeout>

Time allowed for the server to answer the check. If both inter and timeout check are set, then the smaller value of the two is used after the TCP connection is established

When a timeout check is not set, inter starts with the establishment of the TCP connection.

The illustration above describes the settings used depending on the configuration.

Health check logs

By default, HAProxy only logs health checks that trigger a state change from UP to DOWN.

You can log any change in the check status or the server's health by enabling the directive option log-health-checks in the backend or defaults sections.

The logs can show that a server failed occasional checks prior to crashing, for example when:

It fails to return a valid HTTP status
The port starts to reject connections
The server stops responding completely.

Check a TCP Port

To enable a TCP handshake, you use the following parameters:

In the backend section:

option tcp-check (optional)

Highlights the TCP handshake check

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid when the server answers with a SYN/ACK packet.

Perform a TCP handshake on port 80, even though the load-balancer port is 443.

backend be_myapp
        [...]
        option tcp-check
        server srv1 10.0.0.1:443 check port 80
        server srv2 10.0.0.2:443 check port 80

Check an SSL Port

To enable an SSL handshake, you use the following parameters:

In the backend section:

option ssl-hello-chk

Sends an SSLv3 client hello message

On the server line:

`check`	Enables health checking
port	a TCP port configured on the server IP or the `port` parameter

The check is valid if the server answers with a valid SSL server hello message.

Send an SSL hello handshake every 10s; set the server to DOWN after two unsuccessful checks; log any error that occurred:

backend be_myapp
[...]
        option ssl-hello-chk
        option log-health-checks
        default-server inter 10s fall 2
        server srv1 10.0.0.1:443 check
        server srv2 10.0.0.2:443 check

Check an HTTP service

HAProxy can send an HTTP request and analyze the response's status and/or body to validate the service status:

In the backend section, use the following directive:

option httpchk [<method>] [<uri>] [<version>] (mandatory)

`[<method>]` (optional)	HTTP method used to build the HTTP request. If not set, then OPTIONS is used.
`[<uri>]` (optional) requires `<method>`	Sets the URI for the HTTP request. If not set, then / is used. Query strings are allowed.
`[<version>]` (optional) requires `<uri>`	Sets the HTTP version protocol of the request. If not set, HTTP/1.0 is used.

Note

You can send HTTP headers after the <version> string by concatenating them using \r\n and backslashes or spaces. This is useful when sending host headers to probe a virtual host.

http-check disable-on-404

switches the server to administrative status Maintenance when the server answers the health check with a "404". This option has precedence over the option http-check expect. It is useful to switch a server to soft-stop mode and can be triggered at the server layer without any modification in HAProxy.

http-check expect [!] <match> <pattern>

Changes the response validation behavior by setting up a match rule:

[!]

Negate the match result

<match>

Use the matching method to look for <pattern> in the response. Accepted matching methods are

`status`	raw status code comparison
`rstatus`	regex match on status code
`string`	raw sting match in the response's body
`rstring`	regex match in the response's body

<pattern>

Raw information or regex that the <match> method uses

http-check send-state

Adds a X-Haproxy-Server-State HTTP header in the request. This header contains the following information, separated by semi-colons:

Server's operational status (UP, DOWN, NOLB); transitional status (number of failed or successful checks)
String name; the back end and server names separated by a slash '/'
String node; the name of the HAProxy node
String weight; server's weight; '/'; sum of the weight of available servers in the farm
String scur; number of connections on the server; '/' ; the total number of connections on the back end
String qcur; number of requests currently on the server's queue at the HAProxy layer

Example of a send-state line:

X-Haproxy-Server-State:  UP 2/3; name=bck/srv2; node=lb1; weight=1/2; scur=13/22; qcur=0

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server answers with a status code of 2xx or 3xx, unless this behavior is changed by the directive http-check expect.

Simplest HTTP health check on a web server:

backend bk_myapp
        [...]
option httpchk
         server srv1 10.0.0.1:80 check
         server srv2 10.0.0.2:80 check

Equivalent of the configuration above, with all default options:

backend bk_myapp
        [...]
        option httpchk OPTIONS / HTTP/1.0
        http-check expect rstatus (2|3)[0-9][0-9]
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" and consider only the status code "200" as valid:

backend bk_myapp
        [...]
        option httpchk GET /check
        http-check expect status 200
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" and consider all statuses as valid except 5xx:

backend bk_myapp
        [...]
        option httpchk GET /check
        http-check expect ! rstatus ^5
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" and look for the keywork OK in the response's body. It may happen after 20K bytes:

global
        tune.chksize 32768
        [...]
backend bk_myapp
        [...]
        option httpchk GET /check
        http-check expect string OK
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com:

backend bk_myapp
        [...]
        option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com and consider "404" as maintenance mode:

backend bk_myapp
        [...]
        option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
        http-check disable-on-404
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:80 check
        server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com of a TCP based farm that does load-balancing on TLS traffic (HAProxy does not handle SSL in this mode):

backend bk_myapp
        mode tcp
        [...]
        option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:443 check check-ssl
        server srv2 10.0.0.2:443 check check-ssl

Send the request "get /check" to the virtual host www.domain.com of an HTTP farm (HAProxy ciphers the traffic to the server in this mode):

backend bk_myapp
        mode http
        [...]
        option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
        default-server inter 3s fall 3 rise 2
        server srv1 10.0.0.1:443 ssl check
        server srv2 10.0.0.2:443 ssl check

Check an LDAP service

HAProxy can perform simple anonymous LDAPv3 bind checks:

In the backend section:

option ldap-check

Turns on LDAPv3 check

On the server line:

`check`	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server response contains a successful resultCode.

Note

You must configure the LDAP servers according to this check to allow anonymous binding. You can do this with an IP alias on the server side that allows only HAProxy IP addresses to bind to it.

Note

For more advanced LDAP checks, see the section check any service below.

Check an LDAP service on a server's alternative IP address:

backend be_myapp
        [...]
        option ldap-check
        server srv1 10.0.0.1:389 check addr 10.0.0.11
        server srv2 10.0.0.2:389 check addr 10.0.0.12

Check LDAP services on a server's alternative IP address:

backend be_myapp
        [...]
        option ldap-check
        server srv1 10.0.0.1:636 check check-ssl addr 10.0.0.11
        server srv2 10.0.0.2:636 check check-ssl addr 10.0.0.12

Check a MySql Service

Note

This check is compatible with MySql server 3.22 and later.

HAProxy can perform a simple MySql check, either by checking the MySql Handshake packet or testing a full Client Authentication test:

In the backend section:

option mysql-check [user <username>] [post-41]

Turns on MySql check

Without the user option

Performs a MySql handshake

user <username> (optional)

Uses <username> to perform a Client Authentication check

This requires an update in the MySql servers using MySql client software, as shown below:

USE mysql;
INSERT INTO user (Host,User) values ('<ip_of_haproxy>','<username>');
FLUSH PRIVILEGES;

post-41 (optional)

Send checks compatible with MySql server 4.1 and later

Note

If MySql server checks are too frequent and/or if there is too much traffic, the MySql server can block HAProxy. To clear the block, issue a FLUSH HOSTS on the MySql server.

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server response contains a successful resultCode.

Note

For more advanced LDAP checks, see the section check any service below.

Check a MySql service:

backend be_myapp
        [...]
        option mysql-check
        server srv1 10.0.0.1:3306 check
        server srv2 10.0.0.2:3306 check

Check a MySql service using the Client Authentication method for a MySql 5.1 server:

backend be_myapp
        [...]
        option mysql-check user haproxy post-41
        server srv1 10.0.0.1:3306 check
        server srv2 10.0.0.2:3306 check

Check a PgSQL Service

HAProxy can perform a simple PostgreSQL check by sending a StartupMessage.

In the backend section:

`option pgsql-check [user <username>]`	Turns on PostgreSQL check
`user <username>` (optional)	Uses `<username>` to perform a Client Authentication check

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server response contains a successful Authentication request.

Check a PgSQL service:

backend be_pgsql
        [...]
        option pgsql-check
        server srv1 10.0.0.1:5432 check
        server srv2 10.0.0.2:5432 check

Check a PgSql service using the Client Authentication method:

backend be_pgsql
        [...]
        option pgsql-check user haproxy
        server srv1 10.0.0.1:5432 check
        server srv2 10.0.0.2:5432 check

Check a Redis Service

HAProxy can perform a Redis health check by sending the PING command:

In the backend section:

option redis-check

Turns on redis check

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server response contains the string +PONG.

Note

For more advanced LDAP checks, see the section check any service below.

Check a redis service:

backend be_redis
        [...]
        option redis-check
        server srv1 10.0.0.1:6379 check
        server srv2 10.0.0.2:6379 check

Check an SMTP service

HAProxy can perform an SMTP health check:

In the backend section:

option smtpchk [HELO|EHLO] [<domain>]

Turns on the SMTP check

	Without any options, a HELO localhost is sent to the server.
`HELO` or `EHLO` (optional, requires `<domain>`)	Type of SMTP hello command to use: HELO for SMTP and EHLO for ESTMP. Any other value here with default back to HELO
`<domain>` (optional, requires `HELO` or `EHLO`)	Domain name to present to the server

On the server line:

`check`:	Enables health checking
port	A TCP port configured on the server IP or the `port` parameter

The check is valid if the server response code starts with '2'.

Note

For more advanced LDAP checks, see the section check any service below.

Check an SMTP service:

backend be_smtp
        [...]
        option smtp-check
        server srv1 10.0.0.1:25 check
        server srv2 10.0.0.2:25 check

Check an ESMTP service using our haproxytest.check domain:

backend be_smtp
        [...]
        option smtp-check EHLO haproxytest.check
        server srv1 10.0.0.1:25 check
        server srv2 10.0.0.2:25 check

Check a Postfix farm where the proxy-protocol is used:

backend be_smtp
        [...]
        option smtp-check
        server srv1 10.0.0.1:25 check send-proxy
        server srv2 10.0.0.2:25 check send-proxy

Check any service

HAProxy can run health checks in a send / expect manner to allow as many checks as needed to validate a service state. This enables you to configure health checks in a script.

The following directives are available in the backend section:

option tcp-check

Enables and allows tcp-check comment, connect / send / send-binary / expect sequences

tcp-check comment <string>

Adds a comment in the rule set. The <string> is logged by HAProxy to facilitate troubleshooting

tcp-check connect [port <port>] [send-proxy] [ssl]

Establishes a new TCP connection

`port <port>` (optional)	TCP port where the TCP connection is established. If not set, uses the server's line port; if set, uses the port on the configured server
`send-proxy` (optional)	Forces the sending of the proxy protocol over the new connection
`ssl` (optional)	Enables the encryption of the connection

tcp-check expect [!] <match> <pattern>

Method to use to analyze the data sent by the server:

[!]

Negates the match result

<match>

Matching method used to look for <pattern> in the response. Accepted match method are:

`binary`	Raw binary string match in the response's buffer
`string`	Raw sting match in the response's buffer
`rstring`	Regex match in the response's buffer

<pattern>

Raw string or regex used by the <match> method. Spaces must be back slashes. For binary string matching, <pattern> is passed as a series of hexadecimal digits in a even number. Each one represents a byte.

tcp-check send <data>

Specifies the request string to send to the server

tcp-check send-binary <hexadata>

Specifies the request string in hexadecimal be send to the server

Note

Multiple tcp-check send or tcp-check send-binary in a row are concatenated before the are sent out.

On the server line:

`check`:	Enables health checking
port	a TCP port configured on the server IP or the `port` parameter

Check a POP3 service:

backend be_pop
        [...]
        option tcp-check
        tcp-check connect 110
        tcp-check expect string +OK
        server srv1 10.0.0.1:110 check
        server srv2 10.0.0.2:110 check

Check POP3 and POP3S services in a farm where both are load-balanced:

backend be_pop
        [...]
        option tcp-check
        tcp-check comment POP
        tcp-check connect 110
        tcp-check expect string +OK
        tcp-check comment POPs
        tcp-check connect 995
        tcp-check expect string +OK
        server srv1 10.0.0.1 check
        server srv2 10.0.0.2 check

Check in a Redis farm to see which server is the master to select it as valid:

backend be_redis
        [...]
        option tcp-check
        tcp-check send PING\r\n
        tcp-check expect string +PONG
        tcp-check send info replication\r\n
        tcp-check expect string role: master
        tcp-check send QUIT\r\n
        tcp-check expect string +OK
        server srv1 10.0.0.1:6379 check
        server srv2 10.0.0.2:6379 check

When an HTTP service answers with a status code 200, check that it has the string "service up" in the body:

backend be_myapp
        [...]
        option tcp-check
        tcp-check comment "HTTP request"
        tcp-check send GET /check HTTP/1.0\r\n
        tcp-check send Host:  www.mydomain.com\r\n
        tcp-check send \r\n
        tcp-check comment "HTTP response"
        tcp-check send GET /check HTTP/1.0\r\n
        tcp-check expect rstring ^HTTP/1.1 200 Ok
        tcp-check expect string service up
        server srv1 10.0.0.1:8080 check
        server srv2 10.0.0.2:8080 check
backend be_myapp
        [...]
        option tcp-check
        tcp-check send GET /check HTTP/1.0\r\n
        tcp-check send Host: www.mydomain.com\r\n
        tcp-check send \r\n
        tcp-check expect rstring ^HTTP/1.1 200 Ok
        tcp-check expect string service up
        server srv1 10.0.0.1:8080 check
        server srv2 10.0.0.2:8080 check

Forge a binary string to check php-fpm server status:

First, enable the following statements in your php-fpm configuration:

ping.path = /ping
                ping.response = pong

Then configure HAProxy to forge a fast-cgi query to reach the /ping URL and look for the pong keyword in the response:

backend be_phpfpm
        [...]
        option tcp-check
        # FCGI_BEGIN_REQUEST
        tcp-check send-binary   01 # version
        tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
        tcp-check send-binary 0001 # request id
        tcp-check send-binary 0008 # content length
        tcp-check send-binary   00 # padding length
        tcp-check send-binary   00 #
        tcp-check send-binary 0001 # FCGI responder
        tcp-check send-binary 0000 # flags
        tcp-check send-binary 0000 #
        tcp-check send-binary 0000 #
        # FCGI_PARAMS
        tcp-check send-binary   01 # version
        tcp-check send-binary   04 # FCGI_PARAMS
        tcp-check send-binary 0001 # request id
        tcp-check send-binary 0045 # content length
        tcp-check send-binary   03 # padding length:  padding for content % 8 = 0
        tcp-check send-binary   00 #
        tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
        tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
        tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
        tcp-check send-binary 040455534552524F4F54 # USER = ROOT
        tcp-check send-binary 000000 # padding
        # FCGI_PARAMS
        tcp-check send-binary   01 # version
        tcp-check send-binary   04 # FCGI_PARAMS
        tcp-check send-binary 0001 # request id
        tcp-check send-binary 0000 # content length
        tcp-check send-binary   00 # padding length:  padding for content % 8 = 0
        tcp-check send-binary   00 #
        tcp-check expect binary 706f6e67 # pong

Note

The complete binary string can be set up in one line, but the presentation above makes it clearer for later updates.

HAProxy Enterprise 2.0r1 Documentation

Searching HAProxy Enterprise 2.0r1

Performing Health Checks

Understanding the Server's Operational State

Configuring a Health Check

Set the IP address, port, and protocol

Defining health check parameters

Check Interval

Initial check (startup)

Check buffer size

Response validation

Check timeouts

Health check logs

Check a TCP Port

Check an SSL Port

Check an HTTP service

Check an LDAP service

Check a MySql Service

Check a PgSQL Service

Check a Redis Service

Check an SMTP service

Check any service