Before HAProxy forwards traffic to servers, we recommend that you configure it to perform health checks on the service hosted by each server in a farm.

Note

It is important to set up health checks as close as possible to the load-balanced application.

Understanding the Server's Operational State

From an HAProxy point of view, a server can have different operational states.

A server state is defined by the latest check results. The operational states are:

UP

Server is operational

UP - transitionally DOWN

Server is currently considered as operational, but the last check failed. Hence, the server is transitioning to the DOWN state

DOWN - transitionally UP

Server is currently considered as non-operational, but the last check succeeded. Hence, the server is transitioning to the UP state

DOWN

Server is non-operational

The operational state changes are triggered by health check parameters.

HAProxy server state checks diagram

The diagram above shows the different server states and how HAProxy parameters interact with them.

Configuring a Health Check

The minimum configuration for a health check is the check keyword on a server line. In order to run, a health check requires at least an IP address and a TCP port from the server.

Set the IP address, port, and protocol

In HAProxy, all servers configured in a backend get the same health check and must return the same type of response.

A check needs at least the following information:

Information

Default value

Alternative value

IP address

Server IP address

Address provided by server's addr parameter

TCP port

Server port

TCP port provided by server's port parameter

Proxy protocol

Server send-proxy parameter

Follow flag check-send-proxy from the server line

Ciphered

connection Server's ssl parameter

Follow flag check-ssl from the server line

Protocol

TCP handshake

SSL, HTTP, LDAP, MySql, PgSQL, redis, SMTP, generic Send/Expect

Valid response

Depends on the protocol used

Note

When set, addr and port parameters have precedence over the server's IP and port.

Defining health check parameters

Check Interval

All the keywords below apply to the server or default-server directives which you can use to set up a health check frequency based on the server's state:

Parameter

Description

inter

Sets the interval between two consecutive health checks. If not specified, the default value is 2s.

fastinter

Sets the interval between two consecutive health checks when the server is in any of the transition states: UP - transitionally DOWN or DOWN - transitionally UP. If not set, then inter is used.

downinter

Sets the interval between two consecutive health checks when the server is in the DOWN state. If not set, then inter is used.

Initial check (startup)

All the keywords below apply to the global section and can be used to set up HAProxy's behavior when running the first check after startup:

Keyword

Description

max-spread-checks <delay>

When starting up, HAProxy administers the first health checks for a farm over the inter period. When inter is very long, some servers may appear UP after a very long time. This parameter allows the reduction of this initial period to <delay>.

spread-checks <0..50, in percent>

Adds some randomness on the interval delay between two consecutive checks to avoid sending health checks at intervals that are too regular. If not set, the default value is 0.

Check buffer size

Some health checks may need to look for data in the response body. By default, HAProxy reads only the first 16384 bytes of the response and ignores the rest. If you want to search for information above this value, you must set up the tune.chksize parameter in the global section.

Response validation

All the keywords below apply to the server or default-server directives and can be used to tell HAProxy to consider the number of positive or negative valid checks before it changes a server's state:

Keyword

Description

rise <count>

Number of consecutive valid health checks before considering the server as UP. Default value is 2

fall

Number of consecutive invalid health checks before considering the server as DOWN. Default value is 3

Check timeouts

All the keywords below apply to the backend or defaults sections and can be used to tell HAProxy how long to wait for a server response:

Keyword

Description

timeout check <timeout>

Time allowed for the server to answer the check. If both inter and timeout check are set, then the smaller value of the two is used after the TCP connection is established

When a timeout check is not set, inter starts with the establishment of the TCP connection.
HAProxy checks and timeouts

The illustration above describes the settings used depending on the configuration.

Health check logs

By default, HAProxy only logs health checks that trigger a state change from UP to DOWN.

You can log any change in the check status or the server's health by enabling the directive option log-health-checks in the backend or defaults sections.

The logs can show that a server failed occasional checks prior to crashing, for example when:

  • It fails to return a valid HTTP status

  • The port starts to reject connections

  • The server stops responding completely.

Check a TCP Port

To enable a TCP handshake, you use the following parameters:

In the backend section:

option tcp-check (optional)

Highlights the TCP handshake check

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid when the server answers with a SYN/ACK packet.

Perform a TCP handshake on port 80, even though the load-balancer port is 443.

backend be_myapp
    # ...
    option tcp-check
    server srv1 10.0.0.1:443 check port 80
    server srv2 10.0.0.2:443 check port 80

Check an SSL Port

To enable an SSL handshake, you use the following parameters:

In the backend section:

option ssl-hello-chk

Sends an SSLv3 client hello message

On the server line:

check

Enables health checking

port

a TCP port configured on the server IP or the port parameter

The check is valid if the server answers with a valid SSL server hello message.

Send an SSL hello handshake every 10s; set the server to DOWN after two unsuccessful checks; log any error that occurred:

backend be_myapp
    # ...
    option ssl-hello-chk
    option log-health-checks
    default-server inter 10s fall 2
    server srv1 10.0.0.1:443 check
    server srv2 10.0.0.2:443 check

Check an HTTP service

HAProxy can send an HTTP request and analyze the response's status and/or body to validate the service status:

In the backend section, use the following directive:

option httpchk [<method>] [<uri>] [<version>] (mandatory)

[<method>] (optional)

HTTP method used to build the HTTP request. If not set, then OPTIONS is used.

[<uri>] (optional) requires <method>

Sets the URI for the HTTP request. If not set, then / is used. Query strings are allowed.

[<version>] (optional) requires <uri>

Sets the HTTP version protocol of the request. If not set, HTTP/1.0 is used.

Note

You can send HTTP headers after the <version> string by concatenating them using \r\n and backslashes or spaces. This is useful when sending host headers to probe a virtual host.

http-check disable-on-404

switches the server to administrative status Maintenance when the server answers the health check with a "404". This option has precedence over the option http-check expect. It is useful to switch a server to soft-stop mode and can be triggered at the server layer without any modification in HAProxy.

http-check expect [!] <match> <pattern>

Changes the response validation behavior by setting up a match rule:

[!]

Negate the match result

<match>

Use the matching method to look for <pattern> in the response. Accepted matching methods are

status

raw status code comparison

rstatus

regex match on status code

string

raw sting match in the response's body

rstring

regex match in the response's body

<pattern>

Raw information or regex that the <match> method uses

http-check send-state

Adds a X-Haproxy-Server-State HTTP header in the request. This header contains the following information, separated by semi-colons:

  • Server's operational status (UP, DOWN, NOLB); transitional status (number of failed or successful checks)

  • String name; the back end and server names separated by a slash '/'

  • String node; the name of the HAProxy node

  • String weight; server's weight; '/'; sum of the weight of available servers in the farm

  • String scur; number of connections on the server; '/' ; the total number of connections on the back end

  • String qcur; number of requests currently on the server's queue at the HAProxy layer

Example of a send-state line:

X-Haproxy-Server-State:  UP 2/3; name=bck/srv2; node=lb1; weight=1/2; scur=13/22; qcur=0

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server answers with a status code of 2xx or 3xx, unless this behavior is changed by the directive http-check expect.

Simplest HTTP health check on a web server:

backend bk_myapp
    # ...
    option httpchk
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Equivalent of the configuration above, with all default options:

backend bk_myapp
    # ...
    option httpchk OPTIONS / HTTP/1.0
    http-check expect rstatus (2|3)[0-9][0-9]
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" and consider only the status code "200" as valid:

backend bk_myapp
    # ...
    option httpchk GET /check
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" and consider all statuses as valid except 5xx:

backend bk_myapp
    # ...
    option httpchk GET /check
    http-check expect ! rstatus ^5
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" and look for the keywork OK in the response's body. It may happen after 20K bytes:

global
    tune.chksize 32768
    # ...
backend bk_myapp
    # ...
    option httpchk GET /check
    http-check expect string OK
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com:

backend bk_myapp
    # ...
    option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com and consider "404" as maintenance mode:

backend bk_myapp
    # ...
    option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
    http-check disable-on-404
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:80 check
    server srv2 10.0.0.2:80 check

Send the request "get /check" to the virtual host www.domain.com of a TCP based farm that does load-balancing on TLS traffic (HAProxy does not handle SSL in this mode):

backend bk_myapp
    mode tcp
    # ...
    option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:443 check check-ssl
    server srv2 10.0.0.2:443 check check-ssl

Send the request "get /check" to the virtual host www.domain.com of an HTTP farm (HAProxy ciphers the traffic to the server in this mode):

backend bk_myapp
    mode http
    # ...
    option httpchk GET /check "HTTP/1.0\r\nHost: www.domain.com"
    default-server inter 3s fall 3 rise 2
    server srv1 10.0.0.1:443 ssl check
    server srv2 10.0.0.2:443 ssl check

Check an LDAP service

HAProxy can perform simple anonymous LDAPv3 bind checks:

In the backend section:

option ldap-check

Turns on LDAPv3 check

On the server line:

check

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

Note

You must configure the LDAP servers according to this check to allow anonymous binding. You can do this with an IP alias on the server side that allows only HAProxy IP addresses to bind to it.

Note

For more advanced LDAP checks, see the section check any service below.

Check an LDAP service on a server's alternative IP address:

backend be_myapp
    # ...
    option ldap-check
    server srv1 10.0.0.1:389 check addr 10.0.0.11
    server srv2 10.0.0.2:389 check addr 10.0.0.12

Check LDAP services on a server's alternative IP address:

backend be_myapp
    # ...
    option ldap-check
    server srv1 10.0.0.1:636 check check-ssl addr 10.0.0.11
    server srv2 10.0.0.2:636 check check-ssl addr 10.0.0.12

Check a MySql Service

Note

This check is compatible with MySql server 3.22 and later.

HAProxy can perform a simple MySql check, either by checking the MySql Handshake packet or testing a full Client Authentication test:

In the backend section:

option mysql-check [user <username>] [post-41]

Turns on MySql check

Without the user option

Performs a MySql handshake

user <username> (optional)

Uses <username> to perform a Client Authentication check

This requires an update in the MySql servers using MySql client software, as shown below:

USE mysql;
INSERT INTO user (Host,User) values ('<ip_of_haproxy>','<username>');
FLUSH PRIVILEGES;

post-41 (optional)

Send checks compatible with MySql server 4.1 and later

Note

If MySql server checks are too frequent and/or if there is too much traffic, the MySql server can block HAProxy. To clear the block, issue a FLUSH HOSTS on the MySql server.

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

Note

For more advanced LDAP checks, see the section check any service below.

Check a MySql service:

backend be_myapp
    # ...
    option mysql-check
    server srv1 10.0.0.1:3306 check
    server srv2 10.0.0.2:3306 check

Check a MySql service using the Client Authentication method for a MySql 5.1 server:

backend be_myapp
    # ...
    option mysql-check user haproxy post-41
    server srv1 10.0.0.1:3306 check
    server srv2 10.0.0.2:3306 check

Check a PgSQL Service

HAProxy can perform a simple PostgreSQL check by sending a StartupMessage.

In the backend section:

option pgsql-check [user <username>]

Turns on PostgreSQL check

user <username> (optional)

Uses <username> to perform a Client Authentication check

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful Authentication request.

Check a PgSQL service:

backend be_pgsql
    # ...
    option pgsql-check
    server srv1 10.0.0.1:5432 check
    server srv2 10.0.0.2:5432 check

Check a PgSql service using the Client Authentication method:

backend be_pgsql
    # ...
    option pgsql-check user haproxy
    server srv1 10.0.0.1:5432 check
    server srv2 10.0.0.2:5432 check

Check a Redis Service

HAProxy can perform a Redis health check by sending the PING command:

In the backend section:

option redis-check

Turns on redis check

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server response contains the string +PONG.

Note

For more advanced LDAP checks, see the section check any service below.

Check a redis service:

backend be_redis
    # ...
    option redis-check
    server srv1 10.0.0.1:6379 check
    server srv2 10.0.0.2:6379 check

Check an SMTP service

HAProxy can perform an SMTP health check:

In the backend section:

option smtpchk [HELO|EHLO] [<domain>]

Turns on the SMTP check

Without any options, a HELO localhost is sent to the server.

HELO or EHLO (optional, requires <domain>)

Type of SMTP hello command to use: HELO for SMTP and EHLO for ESTMP. Any other value here with default back to HELO

<domain> (optional, requires HELO or EHLO)

Domain name to present to the server

On the server line:

check:

Enables health checking

port

A TCP port configured on the server IP or the port parameter

The check is valid if the server response code starts with '2'.

Note

For more advanced LDAP checks, see the section check any service below.

Check an SMTP service:

backend be_smtp
    # ...
    option smtp-check
    server srv1 10.0.0.1:25 check
    server srv2 10.0.0.2:25 check

Check an ESMTP service using our haproxytest.check domain:

backend be_smtp
    # ...
    option smtp-check EHLO haproxytest.check
    server srv1 10.0.0.1:25 check
    server srv2 10.0.0.2:25 check

Check a Postfix farm where the proxy-protocol is used:

backend be_smtp
    # ...
    option smtp-check
    server srv1 10.0.0.1:25 check send-proxy
    server srv2 10.0.0.2:25 check send-proxy

Check any service

HAProxy can run health checks in a send / expect manner to allow as many checks as needed to validate a service state. This enables you to configure health checks in a script.

The following directives are available in the backend section:

option tcp-check

Enables and allows tcp-check comment, connect / send / send-binary / expect sequences

tcp-check comment <string>

Adds a comment in the rule set. The <string> is logged by HAProxy to facilitate troubleshooting

tcp-check connect [port <port>] [send-proxy] [ssl]

Establishes a new TCP connection

port <port> (optional)

TCP port where the TCP connection is established. If not set, uses the server's line port; if set, uses the port on the configured server

send-proxy (optional)

Forces the sending of the proxy protocol over the new connection

ssl (optional)

Enables the encryption of the connection

tcp-check expect [!] <match> <pattern>

Method to use to analyze the data sent by the server:

[!]

Negates the match result

<match>

Matching method used to look for <pattern> in the response. Accepted match method are:

binary

Raw binary string match in the response's buffer

string

Raw sting match in the response's buffer

rstring

Regex match in the response's buffer

<pattern>

Raw string or regex used by the <match> method. Spaces must be back slashes. For binary string matching, <pattern> is passed as a series of hexadecimal digits in a even number. Each one represents a byte.

tcp-check send <data>

Specifies the request string to send to the server

tcp-check send-binary <hexadata>

Specifies the request string in hexadecimal be send to the server

Note

Multiple tcp-check send or tcp-check send-binary in a row are concatenated before the are sent out.

On the server line:

check:

Enables health checking

port

a TCP port configured on the server IP or the port parameter

Check a POP3 service:

backend be_pop
    # ...
    option tcp-check
    tcp-check connect 110
    tcp-check expect string +OK
    server srv1 10.0.0.1:110 check
    server srv2 10.0.0.2:110 check

Check POP3 and POP3S services in a farm where both are load-balanced:

backend be_pop
    # ...
    option tcp-check
    tcp-check comment POP
    tcp-check connect 110
    tcp-check expect string +OK
    tcp-check comment POPs
    tcp-check connect 995
    tcp-check expect string +OK
    server srv1 10.0.0.1 check
    server srv2 10.0.0.2 check

Check in a Redis farm to see which server is the master to select it as valid:

backend be_redis
    # ...
    option tcp-check
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    tcp-check send info replication\r\n
    tcp-check expect string role: master
    tcp-check send QUIT\r\n
    tcp-check expect string +OK
    server srv1 10.0.0.1:6379 check
    server srv2 10.0.0.2:6379 check

When an HTTP service answers with a status code 200, check that it has the string "service up" in the body:

backend be_myapp
    # ...
    option tcp-check
    tcp-check comment "HTTP request"
    tcp-check send GET /check HTTP/1.0\r\n
    tcp-check send Host:  www.mydomain.com\r\n
    tcp-check send \r\n
    tcp-check comment "HTTP response"
    tcp-check send GET /check HTTP/1.0\r\n
    tcp-check expect rstring ^HTTP/1.1 200 Ok
    tcp-check expect string service up
    server srv1 10.0.0.1:8080 check
    server srv2 10.0.0.2:8080 check
backend be_myapp
    # ...
    option tcp-check
    tcp-check send GET /check HTTP/1.0\r\n
    tcp-check send Host: www.mydomain.com\r\n
    tcp-check send \r\n
    tcp-check expect rstring ^HTTP/1.1 200 Ok
    tcp-check expect string service up
    server srv1 10.0.0.1:8080 check
    server srv2 10.0.0.2:8080 check

Forge a binary string to check php-fpm server status:

First, enable the following statements in your php-fpm configuration:

ping.path = /ping
ping.response = pong

Then configure HAProxy to forge a fast-cgi query to reach the /ping URL and look for the pong keyword in the response:

backend be_phpfpm
    # ...
    option tcp-check
    # FCGI_BEGIN_REQUEST
    tcp-check send-binary   01 # version
    tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
    tcp-check send-binary 0001 # request id
    tcp-check send-binary 0008 # content length
    tcp-check send-binary   00 # padding length
    tcp-check send-binary   00 #
    tcp-check send-binary 0001 # FCGI responder
    tcp-check send-binary 0000 # flags
    tcp-check send-binary 0000 #
    tcp-check send-binary 0000 #
    # FCGI_PARAMS
    tcp-check send-binary   01 # version
    tcp-check send-binary   04 # FCGI_PARAMS
    tcp-check send-binary 0001 # request id
    tcp-check send-binary 0045 # content length
    tcp-check send-binary   03 # padding length:  padding for content % 8 = 0
    tcp-check send-binary   00 #
    tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
    tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
    tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
    tcp-check send-binary 040455534552524F4F54 # USER = ROOT
    tcp-check send-binary 000000 # padding
    # FCGI_PARAMS
    tcp-check send-binary   01 # version
    tcp-check send-binary   04 # FCGI_PARAMS
    tcp-check send-binary 0001 # request id
    tcp-check send-binary 0000 # content length
    tcp-check send-binary   00 # padding length:  padding for content % 8 = 0
    tcp-check send-binary   00 #
    tcp-check expect binary 706f6e67 # pong

Note

The complete binary string can be set up in one line, but the presentation above makes it clearer for later updates.