Application layer health checking

Health checking is the ability to probe a server to ensure the service is up and running.
This is one of the root feature of any load-balancer.

One can probe servers and services at different layer of the OSI model:
  * ARP check (not available in HAProxy)
  * ICMP (ping) check (not available in HAProxy)
  * TCP (handshake) check
  * Application (HTTP, MySql, SMTP, POP, etc…) check

The most representative of the application status the check is, the best.

This means that the best way to check a service is to “speak” the protocol itself.
Unfortunately, it is impossible to write one check per protocol, there are too many protocols and some of them are proprietary and/or binary.
That’s why HAProxy 1.5 now embeds a new health checking method called “tcp-check“. It is very simple and basic “send/expect” probing method where HAProxy can send arbitrary strings and match string or regex comparison on server responses.
Many send and expect can be executed in a row to determine the status of a server.

I’ve already explained how to check redis server and how to balance traffic only to the redis master server of a cluster.

Today’s article introduces a binary protocol widely deployed: fastcgi used by php-fpm.

fastcgi binary ping/pong health check

fastcgi is a binary protocol. It means data on the network are not readable by humans, like HTTP or SMTP.
php-fpm relies on this protocol to treat PHP code. It is common to use HAProxy to load-balance many php-fpm servers for resiliency and scalability.

php-fpm Configuration

Enable a dedicated url to probe and setup the response in your php-fpm configuration:

[sourcecode language=”text”]
ping.path = /ping
ping.response = pong
[/sourcecode]

Which means HAProxy has to send a fastcgi request on /ping and expects the server to response with “pong“.

HAProxy health checking for php-fpm

Add the following tcp-check sequence in your php-fpm backend to probe the /ping url and ensure the server answers a “pong“.
The comment at the end of the line describe the php-cgi protocol fields.

[sourcecode language=”text”]
option tcp-check
# FCGI_BEGIN_REQUEST
tcp-check send-binary 01 # version
tcp-check send-binary 01 # FCGI_BEGIN_REQUEST
tcp-check send-binary 0001 # request id
tcp-check send-binary 0008 # content length
tcp-check send-binary 00 # padding length
tcp-check send-binary 00 #
tcp-check send-binary 0001 # FCGI responder
tcp-check send-binary 0000 # flags
tcp-check send-binary 0000 #
tcp-check send-binary 0000 #
# FCGI_PARAMS
tcp-check send-binary 01 # version
tcp-check send-binary 04 # FCGI_PARAMS
tcp-check send-binary 0001 # request id
tcp-check send-binary 0045 # content length
tcp-check send-binary 03 # padding length: padding for content % 8 = 0
tcp-check send-binary 00 #
tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
tcp-check send-binary 0b055343524950545f4e414d452f70696e67 # SCRIPT_NAME = /ping
tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
tcp-check send-binary 040455534552524F4F54 # USER = ROOT
tcp-check send-binary 000000 # padding
# FCGI_PARAMS
tcp-check send-binary 01 # version
tcp-check send-binary 04 # FCGI_PARAMS
tcp-check send-binary 0001 # request id
tcp-check send-binary 0000 # content length
tcp-check send-binary 00 # padding length: padding for content % 8 = 0
tcp-check send-binary 00 #

tcp-check expect binary 706f6e67 # pong
[/sourcecode]

Note that the whole send string could be written on a single line.

Any protocol???

If some day, you have to do the same type of configuration on a protocol nobody knows, simply capture network traffic of a “hello” sequence using tcpdump.
Then send the tcp payload cpatured using the tcp-check send command and configure the appropriate expect.
And it will work!

Links