Performing Health Checks

Before HAProxy forwards traffic to servers, we recommend that you configure it to perform health checks on the service hosted by each server of a farm.

It is important to set up health checks as close as possible to the load-balanced application.

Understanding the Server’s Operational State

From an HAProxy point of view, a server can have different operational states.

A server state is defined by the latest check results. The operational states are:

  • UP: server is operational
  • UP – transitionally DOWN: server is currently considered as operational, but the last check failed. Hence, the server is transitioning to the DOWN state
  • DOWN – transitionally UP: server is currently considered as non-operational, but the last check succeeded. Hence, the server is transitioning to the UP state
  • DOWN: server is non-operational

The operational state changes are triggered by health check parameters.

The diagram below shows the different server states and how HAProxy parameters interact with them:

Configuring a Health Check

The minimum configuration for a health check is the check keyword on a server line. In order to run, a health check requires at least an IP address and a TCP port from the server.

Setting the IP address, port, and protocol

In HAProxy, all servers configured in a backend get the same health check and must return the same type of response.

A check needs at least the following information:

Information Default value Alternative value
IP address Server’s IP address Address provided by server’s addr parameter
TCP port Server’s port TCP port provided by server’s port parameter
Proxy protocol Server’s send-proxy parameter Follow flag check-send-proxy from the server line
Ciphered connection Server’s ssl parameter Follow flag check-ssl from the server line
Protocol TCP handshake SSL, HTTP, LDAP, MySql, PgSQL, redis, SMTP, generic Send/Expect
Valid response Depends on the protocol used  
When set, addr and port parameters have precedence over the server’s IP and port.

Setting health check parameters

Check Interval

All the keywords below apply to the server or default-server directives which you can use to set up a health check frequency based on the server’s state:

  • inter parameter sets the interval between two consecutive health checks. If not specified, the default value is 2s.
  • fastinter parameter sets the interval between two consecutive health checks when the server is in any of the transition states: UP – transitionally DOWN or DOWN – transitionally UP. If not set, then inter is used.
  • downinter parameter sets the interval between two consecutive health checks when the server is in the DOWN state. If not set, then inter is used.

Initial check (startup)

All the keywords below apply to the global section and can be used to set up HAProxy‘s behavior when running the first check after startup:

  • max-spread-checks <delay>: When starting up, HAProxy administers the first health checks for a farm over the inter period. When inter is very long, some servers may appear UP after a very long time. This parameter allows the reduction of this initial period to <delay>.
  • spread-checks <0..50, in percent>: Adds some randomness on the interval delay between two consecutive checks to avoid sending health checks at intervals that are too regular. If not set, the default value is 0.

Check buffer size

Some health checks may need to look for data in the response body. By default, HAProxy reads only the first 16384 bytes of the response and ignores the rest. If you want to search for information above this value, you must set up the tune.chksize parameter in the global section.

Response validation

All the keywords below apply to the server or default-server directives and can be used to tell HAProxy to consider the number of positive or negative valid checks before it changes a server’s state:

  • rise <count>: number of consecutive valid health checks before considering the server as UP. Default value is 2
  • fall: number of consecutive invalid health checks before considering the server as DOWN. Default value is 3

Check timeouts

All the keywords below apply to the backend or defaults sections and can be used to tell HAProxy how long to wait for a server response:

  • timeout check <timeout>: time allowed for the server to answer the check. If both inter and timeout check are set, then the smaller value of the two is used after the TCP connection is established

When a timeout check is not set, inter starts with the establishment of the TCP connection.

The illustration below describes the settings used depending on the configuration:

Health check logs

By default, HAProxy only logs health checks that trigger a state change from UP to DOWN.

You can log any change in the check status or the server’s health by enabling the directive option log-health-checks in the backend or defaults sections.

The logs can show that a server failed occasional checks prior to crashing; when it failed to return a valid HTTP status; when the port started to reject connections; and when the server stopped responding completely.

Checking a TCP Port

To enable a TCP handshake, you use the following parameters:

– In the backend section:

  • option tcp-check (optional): Highlights the TCP handshake check

– On the server line:

  • check: Enables health checking
  • a TCP port configured on the server IP or the port parameter

The check is valid when the server answers with a SYN/ACK packet.

Example

Perform a TCP handshake on port 80, even though the load-balancer port is 443:

backend be_myapp
	[...]
	option tcp-check
	server srv1 10.0.0.1: 443 check port 80
	server srv2 10.0.0.2: 443 check port 80

Checking an SSL Port

To enable an SSL handshake, you use the following parameters:

– In the backend section:

  • option ssl-hello-chk: Sends an SSLv3 client “hello” message

– On the server line:

  • check: Enables health checking
  • a TCP port configured on the server IP or the port parameter

The check is valid if the server answers with a valid SSL server “hello” message.

Example

Send an SSL “hello” handshake every 10s; set the server to DOWN after two unsuccessful checks; log any error that occurred:

backend be_myapp
[...]
	option ssl-hello-chk
	option log-health-checks	
	default-server inter 10s fall 2
	server srv1 10.0.0.1: 443 check
	server srv2 10.0.0.2: 443 check

Checking an HTTP service

HAProxy can send an HTTP request and analyze the response’s status and/or body to validate the service status:

– In the backend section, use the following directive:

  • option httpchk [<method>] [<uri>] [<version>] (mandatory)
    • [<method>] (optional): HTTP method used to build the HTTP request. If not set, then OPTIONS is used.
    • [<uri>](optional, requires <method>): sets the URI for the HTTP request. If not set, then / is used. Query strings are allowed.
    • [<version>](optional, requires <uri>): sets the HTTP version protocol of the request. If not set, HTTP/1.0 is used.
You can send HTTP headers after the <version> string by concatenating them using rn and backslashes or spaces. This is useful when sending host headers to probe a virtual host.
  • http-check disable-on-404: switches the server to administrative status Maintenance when the server answers the health check with a “404”. This option has precedence over the oprion http-check expect. It is useful to switch a server to soft-stop mode and can be triggered at the server layer without any modification in HAProxy.
  • http-check expect [!] <match> <pattern>: changes the response validation behavior by setting up a match rule:
    • [!]: negate the match result
    • <match>: use the matching method to look for <pattern> in the response. Accepted matching methods are:
      • status: raw status code comparison
      • rstatus: regex match on status code
      • string: raw sting match in the response’s body
      • rstring: regex match in the response’s body
    • <pattern>: raw information or regex that the <match> method uses
  • http-check send-state: adds a X-Haproxy-Server-State HTTP header in the request. This header contains the following information, separated by semi-colons:
    • Server’s operational status (UP, DOWN, NOLB); transitional status (number of failed or successful checks)
    • string name ; the back end and server names separated by a slash ‘/’
    • string node ; the name of the HAProxy node
    • string weight ; server’s weight; ‘/’; sum of the weight of available servers in the farm
    • string scur ; number of connections on the server; ‘/’ ; the total number of connections on the back end
    • string qcur ; number of requests currently on the server’s queue at the HAProxy layer
  • Example of a send-state line:
X-Haproxy-Server-State:  UP 2/3; name=bck/srv2; node=lb1; weight=1/2; scur=13/22; qcur=0

– on the server line:

  • check: Enables health checking
  • a TCP port configured on the server IP or the port parameter

The check is valid if the server answers with a status code of 2xx or 3xx, unless this behavior is changed by the directive http-check expec.

Examples

1. Simplest HTTP health check on a web server:

backend bk_myapp
	[...]
option httpchk
	 server srv1 10.0.0.1: 80 check
	 server srv1 10.0.0.1: 80 check

2. Equivalent of the configuration above, with all default options:

backend bk_myapp
	[...]
	option httpchk OPTIONS / HTTP/1.0
	http-check expect rstatus (2|3)[0-9][0-9]
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

3. Send the request “get /check” and consider only the status code “200” as valid:

backend bk_myapp
	[...]
	option httpchk get /check
	http-check expect status 200
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

4. Send the request “get /check” and consider all statuses as valid except 5xx:

backend bk_myapp
	[...]
	option httpchk get /check
	http-check expect ! rstatus ^5
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

5. Send the request “get /check” and look for the keywork OK in the response’s body. It may happen after 20K bytes:

global
	tune.chksize 32768

	[...]

backend bk_myapp
	[...]		
	option httpchk get /check
	http-check expect string OK
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

6. Send the request “get /check” to the virtual host www.domain.com:

backend bk_myapp
	[...]
	option httpchk get /check HTTP/1.0rnHost:  www.domain.com
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

7. Send the request “get /check” to the virtual host www.domain.com and consider “404” as maintenance mode:

backend bk_myapp
	[...]
	option httpchk get /check HTTP/1.0rnHost:  www.domain.com
	http-check disable-on-404
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 80 check
	server srv1 10.0.0.1: 80 check

8. Send the request “get /check” to the virtual host www.domain.com of a TCP based farm that does load-balancing on TLS traffic (HAProxy does not handle SSL in this mode):

backend bk_myapp
	mode tcp
	[...]
	option httpchk get /check HTTP/1.0rnHost:  www.domain.com
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 443 check check-ssl
	server srv1 10.0.0.1: 443 check check-ssl

9. Send the request “get /check” to the virtual host www.domain.com of an HTTP farm (HAProxy ciphers the traffic to the server in this mode):

backend bk_myapp
	mode http
	[...]
	option httpchk get /check HTTP/1.0rnHost:  www.domain.com
	default-server inter 3s fall 3 rise 2
	server srv1 10.0.0.1: 443 ssl check
	server srv1 10.0.0.1: 443 ssl check

Checking an LDAP service

HAProxy can perform simple anonymous LDAPv3 bind checks:

– In the backend section:

  • option ldap-check: Turns on LDAPv3 check

– On the server line:

  • check: enables health checking
  • a TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

You must configure the LDAP servers according to this check to allow anonymous binding. You can do this with an IP alias on the server side that allows only HAProxy IP addresses to bind to it.
For more advanced LDAP checks, see the section generic Send/Expect below.

Examples

1. Check an LDAP service on a server’s alternative IP address:

backend be_myapp
	[...]
	option ldap-check
	server srv1 10.0.0.1: 389 check addr 10.0.0.11
	server srv2 10.0.0.2: 389 check addr 10.0.0.12

2. Check LDAP services on a server’s alternative IP address:

backend be_myapp
	[...]
	option ldap-check
	server srv1 10.0.0.1: 636 check check-ssl addr 10.0.0.11
	server srv2 10.0.0.2: 636 check check-ssl addr 10.0.0.12

Checking a MySql Service

This check is compatible with MySql server 3.22 and later.

HAProxy can perform a simple MySql check, either by checking the MySql Handshake packet or testing a full Client Authentication test:

  • In the backend section:
  • option mysql-check [user <username>] [post-41]: Turns on MySql check
    • without the user option: performs a MySql handshake
    • user <username> (optional): uses <username> to perform a Client Authentication check
      This requires an update in the MySql servers using MySql client software, as shown below:
      USE mysql;
      INSERT INTO user (Host,User) values ('<ip_of_haproxy>','<username>');
      FLUSH PRIVILEGES;
  • post-41 (optional): Send checks compatible with MySql server 4.1 and later
If MySql server checks are too frequent and/or if there is too much traffic, the MySql server can block HAProxy. To clear the block, issue a FLUSH HOSTS on the MySql server.

– On the server line:

  • check: enables health checking
  • a TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

For more advanced LDAP checks, see the section generic Send/Expect below.

Examples

1. Check a MySql service:

backend be_myapp
	[...]
	option mysql-check
	server srv1 10.0.0.1: 3306 check
	server srv2 10.0.0.2: 3306 check

2. Check a MySql service using the Client Authentication method for a MySql 5.1 server:

backend be_myapp
	[...]
	option mysql-check user haproxy post-41
	server srv1 10.0.0.1: 3306 check
	server srv2 10.0.0.2: 3306 check

Checking a PgSQL Service

HAProxy can perform a simple PostgreSQL check, by sending a StartupMessage.

  • In the backend section:
    • option pgsql-check [user <username>]: Turns on PostgreSQL check
    • user <username>(optional): Uses <username> to perform a Client Authentication check
  • On the server line:
    • check: enables health checking
    • a TCP port configured on the server IP or the port parameter

The check is valid if the server response contains a successful Authentication request.

Examples

1. Check a PgSQL service:

backend be_pgsql
	[...]
	option pgsql-check
	server srv1 10.0.0.1: 5432 check
	server srv2 10.0.0.2: 5432 check

2. Check a PgSql service using the Client Authentication method:

backend be_pgsql
	[...]
	option pgsql-check user haproxy
	server srv1 10.0.0.1: 5432 check
	server srv2 10.0.0.2: 5432 check

Checking a Redis Service

HAProxy can perform a Redis health check by sending the PING command:

  • In the backend section:
    • option redis-check: Turns on redis check
  • On the server line:
    • check: enables health checking
    • a TCP port configured on the server IP or the port parameter

The check is valid if the server response contains the string +PONG.

For more advanced LDAP checks, see the section generic Send/Expect below.

Example

Check a redis service:

backend be_redis
	[...]
	option redis-check
	server srv1 10.0.0.1: 6379 check
	server srv2 10.0.0.2: 6379 check

Checking an SMTP service

HAProxy can perform an SMTP health check:

  • In the back end section:
    • option smtpchk [HELO|EHLO] [<domain>]: Turns on the SMTP check
    • Without any options, a HELO localhost is sent to the server.
    • HELO or EHLO (optional, requires<domain> ): type of SMTP hello command to use: HELO for SMTP and EHLO for ESTMP. Any other value here with default back to HELO
    • <domain> (optional, requiresHELO or EHLO): domain name to present to the server
  • on the server line:
    • check: Enables health checking
    • a TCP port configured on the server IP or the port parameter

The check is valid if the server response code starts with ‘2’.

For more advanced LDAP checks, see the section generic Send/Expect below.

Examples

Check an SMTP service:

backend be_smtp
	[...]
	option smtp-check
	server srv1 10.0.0.1: 25 check
	server srv2 10.0.0.2: 25 check

2. Check an ESMTP service using our haproxytest.check domain

backend be_smtp
	[...]
	option smtp-check EHLO haproxytest.check
	server srv1 10.0.0.1: 25 check
	server srv2 10.0.0.2: 25 check

3. Check a Postfix farm where the proxy-protocol is used:

backend be_smtp
	[...]
	option smtp-check
	server srv1 10.0.0.1: 25 check send-proxy
	server srv2 10.0.0.2: 25 check send-proxy

Checking any service

HAProxy can run health checks in a send / expect manner to allow as many checks as needed to validate a service state. This enables you to configure health checks in a script.

The following directives are available in the backend section:

  • option tcp-check: Enables and allows tcp-check comment, connect / send / send-binary / expect sequences
  • tcp-check comment <string>: Adds a comment in the rule set. The <string> is logged by HAProxy to facilitate troubleshooting
  • tcp-check connect [port <port>] [send-proxy] [ssl]: Establishes a new TCP connection
    • port <port> (optional): TCP port where the TCP connection is established. If not set, uses the server’s line port; if set, uses the port on the configured server
    • send-proxy(optional): Forces the sending of the proxy protocol over the new connection
    • ssl (optional): Enables the encryption of the connection
  • tcp-check expect [!] <match> <pattern>: method to use to analyze the data sent by the server:
    • [!]: Negates the match result
    • <match>: Matching method used to look for <pattern> in the response. Accepted match method are:
      • binary: Raw binary string match in the response’s buffer
      • string: Raw sting match in the response’s buffer
      • rstring: Regex match in the response’s buffer
    • <pattern>: Raw string or regex used by the <match> method. Spaces must be back slashes. For binary string matching, <pattern> is passed as a series of hexadecimal digits in a even number. Each one represents a byte.
  • tcp-check send <data>: Specifies the request string to send to the server
  • tcp-check send-binary <hexadata>: Specifies the request string in hexadecimal be send to the server
Multiple tcp-check send or tcp-check send-binary in a row are concatenated before the are sent out.
  • on the server line:
    • check: Enables health checking
    • a TCP port configured on the server IP or the port parameter

Examples

1. Check a POP3 service:

backend be_pop
	[...]
	option tcp-check
	tcp-check connect 110
	tcp-check expect string +OK
	server srv1 10.0.0.1: 110 check
	server srv2 10.0.0.2: 110 check

2. Check POP3 and POP3S services in a farm where both are load-balanced:

backend be_pop
	[...]
	option tcp-check
	tcp-check comment POP	
	tcp-check connect 110
	tcp-check expect string +OK
	tcp-check comment POPs
	tcp-check connect 995
	tcp-check expect string +OK
	server srv1 10.0.0.1 check
	server srv2 10.0.0.2 check

3. Check in a Redis farm to see which server is the master to select it as valid:

backend be_redis
	[...]
	option tcp-check
	tcp-check send PINGrn
	tcp-check expect string +PONG
	tcp-check send info replicationrn
	tcp-check expect string role: master
	tcp-check send QUITrn
	tcp-check expect string +OK
	server srv1 10.0.0.1: 6379 check
	server srv2 10.0.0.2: 6379 check

4. When an HTTP service answers with a status code 200, check that it has the string “service up” in the body:

backend be_myapp
	[...]
	option tcp-check
	tcp-check comment "HTTP request"
	tcp-check send GET /check HTTP/1.0rn
	tcp-check send Host:  www.mydomain.comrn
	tcp-check send rn
	tcp-check comment "HTTP response"
	tcp-check send GET /check HTTP/1.0rn
	tcp-check expect rstring ^HTTP/1.1 200 Ok
	tcp-check expect string service up
	server srv1 10.0.0.1: 8080 check
	server srv2 10.0.0.2: 8080 check

5. Forge a binary string to check php-fpm server status:

  • First, enable the following statements in your php-fpm configuration:
ping.path = /ping
ping.response = pong
  • Then configure HAProxy to forge a fast-cgi query to reach the /ping URL and look for the pong keyword in the response:
backend be_phpfpm
	[...]
	option tcp-check
	# FCGI_BEGIN_REQUEST
	tcp-check send-binary   01 # version
	tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
	tcp-check send-binary 0001 # request id
	tcp-check send-binary 0008 # content length
	tcp-check send-binary   00 # padding length
	tcp-check send-binary   00 #
	tcp-check send-binary 0001 # FCGI responder
	tcp-check send-binary 0000 # flags
	tcp-check send-binary 0000 #
	tcp-check send-binary 0000 #
	# FCGI_PARAMS
	tcp-check send-binary   01 # version
	tcp-check send-binary   04 # FCGI_PARAMS
	tcp-check send-binary 0001 # request id
	tcp-check send-binary 0045 # content length
	tcp-check send-binary   03 # padding length:  padding for content % 8 = 0
	tcp-check send-binary   00 #
	tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
	tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
	tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
	tcp-check send-binary 040455534552524F4F54 # USER = ROOT
	tcp-check send-binary 000000 # padding
	# FCGI_PARAMS
	tcp-check send-binary   01 # version
	tcp-check send-binary   04 # FCGI_PARAMS
	tcp-check send-binary 0001 # request id
	tcp-check send-binary 0000 # content length
	tcp-check send-binary   00 # padding length:  padding for content % 8 = 0
	tcp-check send-binary   00 #
	tcp-check expect binary 706f6e67 # pong
The complete binary string can be set up in one line, but the presentation above makes it clearer for later updates.

Using External Commands for Health Checks

HAProxy can run an external command in binary or script to perform a health check. When doing this, it needs to fork itself as a new process.

If you use chroot, be sure that the command and all of its dependencies are available in chroot.

The following directives are available:

  • In the global section:
    • external-check: Allows the use of the external check feature. It is disabled by default for security purposes.
  • In the defaults, backend, or listen section:
    • option external-check: Enables the use of an external command to perform health check
    • external-check command <command>: Name of the command to run.

    The arguments below are passed to the command:

    <proxy_address> <proxy_port> <server_address> <server_port>

    where:

    • <proxy_address> and <proxy_port> are set only when this directive is configured in a listen section.
      • Their values are derived from the first listener (bind) that is either IPv4, IPv6 or a UNIX socket.
      • In the case of a UNIX socket listener, the <proxy_address> is the path of the socket, and the <proxy_port> is set to NOT_USED.
      • In a back end section where there is no bind, both are set to NOT_USED.
    • <server_address> and <server_port> are set to the server address and port dedicated to health checking

    Other information is provided through the environment variable <command> which you can use to run a more accurate health check sequence:

    • HAPROXY_PROXY_ADDR: first bind address, available when used in a listen section
    • HAPROXY_PROXY_ID: back end UUID
    • HAPROXY_PROXY_NAME: back end name
    • HAPROXY_PROXY_PORT: first bind port, available when used in a listen section
    • HAPROXY_SERVER_ADDR: server address
    • HAPROXY_SERVER_CURCONN: current active connections on the server
    • HAPROXY_SERVER_ID: server UUID
    • HAPROXY_SERVER_MAXCONN: server’s maxconn
    • HAPROXY_SERVER_NAME: server name
    • HAPROXY_SERVER_PORT: server port (empty for a UNIX socket)
    • PATH: PATH environment variable set by external-check path <path>

    The <command> exit code is used to determine the server’s operational status:

    • 0: check is considered as positive
    • any other value is considered as a failure
  • In the external-check path <path>: set the PATH environment variable of <command> to <path>. Default path is “”.

Example

Run a bash script that needs simple commands like ping to perform health checking:

global
	# chroot /var/empty  # disable chroot
	external-check
	[...]
	backend b_myapp
	option external-check
	external-check path "/usr/bin: /bin"
	external-check command /etc/hapee-1.6/scripts/check.sh"