Over the past 15 years, HAProxy has become well known for its reliability, superior performance, extensible features, and advanced security. It is relatively less known that one of HAProxy’s core building blocks is the Runtime API which provides very powerful dynamic configuration capabilities with no service reloads or restarts.

The early improvements to the Runtime API’s dynamic configuration capabilities were driven by requests from advanced HAProxy users such as Airbnb and Yelp through their development of SmartStack, a pioneering automated service discovery and registration platform. Since then, we have been strongly committed to evolving the Runtime API based on feedback from our users, as exemplified by our recent blog post “Dynamic Scaling for Microservices with the HAProxy Runtime API” and the release of HAProxy version 1.8.

In this blog post we are going to take you on a tour of the HAProxy Runtime API and some of its key features, such as the ability to dynamically configure backend servers, maps, ACLs, stick tables, and TLS ticket keys. These features allow for improved integration with service discovery tools and orchestration systems, enhance geolocation functionality, enable adaptive security policies, increase the efficiency of SSL/TLS clusters, and much more.

Getting Started

The HAProxy Runtime API traces its origins back to our wishes to create a complete configuration and statistics API for HAProxy, whose commands would all take effect immediately, during runtime. One of our early features in this API was of course the ability to retrieve detailed real-time statistics. Also, unlike typical newer APIs which only support HTTP, the HAProxy Runtime API was always accessible through TCP and Unix sockets. That is why we sometimes still refer to it as the HAProxy “stats socket” or just “socket”, and why the configuration directive for enabling the Runtime API bears the same name.

The Runtime API is enabled in the HAProxy configuration by using the following example:

    global
       ...
       stats socket ipv4@127.0.0.1:9999 level admin
       stats socket /var/run/hapee-lb.sock mode 666 level admin
       stats timeout 2m
       ...

If you are not using HAProxy Enterprise Edition, the default Unix socket name in your installation might be named “/var/run/haproxy.sock” instead of “/var/run/hapee-lb.sock”.

Then, for testing or executing commands interactively, the Runtime API can be conveniently accessed using the interactive command prompt provided by “socat”:

$ socat readline /var/run/hapee-lb.sock
$ socat readline tcp4-connect:127.0.0.1:9999

> help

Please note that for the above to work, your version of “socat” needs to be compiled with GNU Readline support. This could be verified by running “socat -V | grep READLINE”. If the output of “socat -V” doesn’t mention “READLINE”, or it mentions “undef READLINE”, or running the actual command produces an error “unknown device/address “readline””, it means you have a version without readline support.

For accessing the Runtime API from scripts, or as an alternative to the interactive use shown above, the following command could be used:

$ echo "help" | socat stdio /var/run/hapee-lb.sock
$ echo "help" | socat stdio tcp4-connect:127.0.0.1:9999

And finally, as an alternative to “socat” altogether, command “nc” from the package “netcat” could be used. In that case, please use the”netcat-openbsd” version of “netcat”; it also supports the option “nc -U” for connecting to Unix domain sockets.

The best way to start with the Runtime API is to execute a simple request returning a list of all available commands. This is done by sending “help” or any unknown command to the API. Here is the actual output from “help”, executed on HAProxy version 1.8-dev2. If your HAProxy is missing any particular commands or options, please make sure you are using a recent HAProxy or HAPEE release.

      > help

      help           : this message
      prompt         : toggle interactive mode with prompt
      quit           : disconnect
      show errors    : report last request and response errors for each proxy
      clear counters : clear max statistics counters (add 'all' for all counters)
      show info      : report information about the running process
      show stat      : report counters for each proxy and server
      show schema json : report schema used for stats
      disable agent  : disable agent checks (use 'set server' instead)
      disable health : disable health checks (use 'set server' instead)
      disable server : disable a server for maintenance (use 'set server' instead)
      enable agent   : enable agent checks (use 'set server' instead)
      enable health  : enable health checks (use 'set server' instead)
      enable server  : enable a disabled server (use 'set server' instead)
      set maxconn server : change a server's maxconn setting
      set server     : change a server's state, weight or address
      get weight     : report a server's current weight
      set weight     : change a server's weight (deprecated)
      show sess [id] : report the list of current sessions or dump this session
      shutdown session : kill a specific session
      shutdown sessions server : kill sessions on a server
      clear table    : remove an entry from a table
      set table [id] : update or create a table entry's data
      show table [id]: report table usage stats or dump this table's contents
      disable frontend : temporarily disable specific frontend
      enable frontend : re-enable specific frontend
      set maxconn frontend : change a frontend's maxconn setting
      show servers state [id]: dump volatile server information (for backend )
      show backend   : list backends in the current running config
      shutdown frontend : stop a specific frontend
      set dynamic-cookie-key backend : change a backend secret key for dynamic cookies
      enable dynamic-cookie backend : enable dynamic cookies on a specific backend
      disable dynamic-cookie backend : disable dynamic cookies on a specific backend
      show stat resolvers [id]: dumps counters from all resolvers section and
                              associated name servers
      set maxconn global : change the per-process maxconn setting
      set rate-limit : change a rate limiting value
      set timeout    : change a timeout setting
      show env [var] : dump environment variables known to the process
      show cli sockets : dump list of cli sockets
      add acl        : add acl entry
      clear acl  : clear the content of this acl
      del acl        : delete acl entry
      get acl        : report the patterns matching a sample for an ACL
      show acl [id]  : report available acls or dump an acl's contents
      add map        : add map entry
      clear map  : clear the content of this map
      del map        : delete map entry
      get map        : report the keys and values matching a sample for a map
      set map        : modify map entry
      show map [id]  : report available maps or dump a map's contents
      show pools     : report information about the memory pools usage

Retrieving Statistics

For a light introduction before going into the Runtime API’s more powerful commands, let’s demonstrate the method for obtaining real-time statistics from HAProxy.

(One of the already well known methods for retrieving real-time statistics is through the HAProxy’s built-in, HTML web page. These statistics, formatted in a simple and straightforward way, can be obtained after configuring the “stats uri” in HAProxy and visiting the statistics URL.)

The same complete and raw information can also be obtained through the HAProxy Runtime API. The API command is named “show stat”:

> show stat

The command’s output will be provided in CSV format. Support for JSON output has been included in HAProxy version 1.8 and HAProxy Enterprise Edition version 1.7r2.

The statistics output contains more than 80 different metrics. To quickly convert it to a shorter and human readable output, we could use standard command line tools. Here is an example that shows a chosen subset of data and refreshes it every 2 seconds:

$ watch 'echo "show stat" | socat stdio /var/run/hapee-lb.sock | cut -d "," -f 1-2,5-10,34-36 | column -s, -t'

# pxname  svname    scur  smax  slim  stot  bin    bout   rate  rate_lim  rate_max
fe_main   FRONTEND  10    10    7000  83    6347   15476  0     0         10
be_stats  BACKEND   0     0     700   0     0      0      0     0
be_app    websrv1   0     0     0     0     0      0      0
be_app    websrv2   0     0     0     0     0      0      0
be_app    BACKEND   0     0     700   0     0      0      0     0
be_other  websrv1   0     10    322   6347  15476  14     20
be_other  BACKEND   10    10    700   83    6347   15476  0     10

By using the command “cut” above, we have narrowed the selection down to the fields useful for quick and summarized monitoring:

  • scur: current number of sessions
  • smax: highest number of sessions seen since HAProxy was reloaded/restarted or had its statistics reset
  • slim: configured upper limit on the number of sessions
  • stot: cumulative number of sessions processed so far
  • bin: bytes in
  • bout: bytes out
  • rate: average number of sessions per second, calculated based on the 1 second
  • rate_lim: configured upper limit on new sessions per second
  • rate_max: highest number of new sessions per second seen since HAProxy was reloaded/restarted or had its statistics reset

For a listing and description of all the fields available in the “show stat” output, please refer to the HAProxy Management Guide, section 9.1.

Dynamically Scaling Backend Servers

The topic of using the HAProxy Runtime API for dynamically scaling backend servers was covered in one of our previous blog posts titled “Dynamic Scaling for Microservices with the HAProxy Runtime API“. Please refer to it for the complete treatment of the subject, especially for use in microservices environments where the rate of changes is very high and where configuration updates must be done during runtime with absolutely no impact on user experience. A brief summary is included here for completeness.

The Runtime API allows changing backend server addresses, ports, weights, and states, and enabling and disabling backends.

The following configuration directives help us implement dynamic scaling using the Runtime API:

First, we could use “server templates” to quickly define template/placeholder slots for up to n backend servers:

backend be_template
  server-template websrv 1-100 192.168.122.1:8080 check disabled

This configuration is equal in effect to manually writing out “server websrvX 192.168.122.1:8080 check disabled” 100 times, but it automatically replaces X with a number incrementing from 1 to 100, inclusive. The servers are added in the disabled state, and it is expected that your server template range (“1-100”) will be larger than the number of servers you currently have to allow for runtime/dynamic scaling to up to n configured backend servers.

Then, after the backend servers are in place (either by using server-template or defining them manually), we could use the Runtime API to configure or update the servers. For example, we could update the IP address and port of ‘websrv1’, and change it from a ‘disabled’ to a ‘ready’ state:

> set server be_template/websrv1 addr 192.168.122.42 port 8080

> set server be_template/websrv1 state ready

In addition, we are also going to ensure that any runtime changes are written out to the state file, and that the state file is loaded on reload/restart:

[global]
  server-state-file /etc/hapee/haproxy.state

[defaults]
  load-server-state-from-file global

Please note that the state file is not updated automatically. To save state, you would run echo “show servers state” | socat stdio /var/run/hapee-lb.sock > /etc/hapee/haproxy.state after making changes to the states and before invoking an HAProxy reload or restart.

For the complete treatment of the subject, please refer to our blog post titled “Dynamic Scaling for Microservices with the HAProxy Runtime API“.

Updating Maps

Maps in HAProxy are a general mechanism for converting sets of input values onto corresponding output values. Maps are used in numerous cases, including for converting HTTP “Host” header values into backend (application) names and for mapping ranges of IP addresses to geographical locations. In the blog post titled “Web Application Name to Backend Mapping in HAProxy“, we have shown practical examples of using the map files and we have mentioned the standalone tool lb-update that can be used to periodically and automatically update map and ACL files. A similar functionality could be obtained by using the HAProxy Runtime API.

The process of updating map contents via the HAProxy Runtime API is simple: we are going to list maps and map contents, identify the entries we want to modify, and then perform the actual modifications. Modifications to the existing entries in the Runtime API generally consist of of two steps – first adding a new value, then deleting the old one to complete the transition.

A complete session for modifying the contents of a map could look like the following:

# List maps
> show map
# id (file) description 
-1 (/etc/hapee/domain2backend.map) pattern loaded from file '/etc/hapee/domain2backend.map' used by map at file '/etc/hapee/hapee-lb.cfg' line 64
-1 (/etc/hapee/domain2redirect.map) pattern loaded from file '/etc/hapee/domain2redirect.map' used by map at file '/etc/hapee1/hapee-lb.cfg' line 65


# Display contents of a particular map
> show map /etc/hapee/domain2backend.map

0x2163350  app1.domain1.com  bk_app1
0x21633d0  app1.domain2.com  bk_app1
0x2163450  app2.domain1.com  bk_app2
0x21634d0  app2.domain2.com  bk_app2
0x2163550  app.domain.com    bk_app1


# Replace the last entry "app.domain.com bk_app1" with "app.domain.com bk_app2"
# Perform the replacement by adding a new value, then deleting the old one
> add  map /etc/hapee/domain2backend.map app.domain.com bk_app2
> del  map /etc/hapee/domain2backend.map #0x2163550

# Verify new contents of the map
> show map /etc/hapee/domain2backend.map

0x2163350  app1.domain1.com  bk_app1
0x21633d0  app1.domain2.com  bk_app1
0x2163450  app2.domain1.com  bk_app2
0x21634d0  app2.domain2.com  bk_app2
0x2163550  app.domain.com    bk_app2

Updating GeoIP Databases

GeoIP databases map IP address ranges to geographical locations. Such databases can be used in HAProxy for different purposes. Often times, they are used for performing GeoIP lookups natively within HAProxy and serving the data to backend servers. The backend servers can then depend on the information being available as part of incoming requests, requiring no specific GeoIP code nor causing any slowdowns in the application code itself.

More information about using GeoIP with HAProxy can be found in one of our previous blog posts titled “Using GeoIP Databases with HAProxy“. Another common use case for GeoIP is to include the client country code in the HAProxy logs. This could be achieved by using the following log-format directive:

log-format "%ci:%cp_%[src,map_ip(/etc/hapee/ip-country.lst)] [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

The above line will use the map_ip converter in order to get the country code from the map file “ip-country.lst”. Logs based on this format will then look like the following:

192.168.122.1:51000_FR [20/Nov/2017:19:34:11.688] fe_main be_app/websrv1 0/0/0/0/0 200 5720 - - --NI 1/1/0/1/0 0/0 "GET / HTTP/1.1"

Now, in terms of updating the GeoIP databases, let’s assume we have the following new GeoIP entries in a map file:

2.0.0.0/12 FR
2.16.0.0/24 FR
2.16.2.0/23 FR
2.16.10.0/24 FR

We could easily add them to the running configuration using the Runtime API command “add map”. Also, instead of adding entries one by one, we are going to use a little bit of bash shell scripting to automate the data import:

# Loop through the map entries on disk and add them to the running configuration
$ IFS=$'\n'
$ for ip_country in $(cat /tmp/more-ip-countries.lst); do
  echo "add map /tmp/ip-country.lst $ip_country"
done | socat stdio  /var/run/hapee-lb.sock

Updating Stick Tables

Stick tables are yet another very powerful and very applicable, but simple to use feature in HAProxy. Stick tables allow keeping track of arbitrary client information across requests, so they are used for purposes such as creating “sticky” sessions, implementing security protections, setting limits, and maintaining counters and lists.

Stick tables (along with timers, schedulers, ACLs, and many other features and tools) are based on high-performance Elastic Binary Trees, invented by our HAProxy Technologies CTO, Willy Tarreau. Ebtrees allow these tables to contain millions of records while maintaining extremely fast speed of access. We have written about stick tables and their uses in many of our previous blog posts, including:

The Runtime API allows stick table entries to be added, removed, searched, and updated during runtime. In the following example, we are going to create a stick table of 1 million entries to keep track of the request rates coming from individual client IPs. We will impose a limit of a maximum 5 requests per second, and the IP addresses which keep sending requests with a rate above that limit over a period of 10 seconds will be denied further requests for a period of 15 minutes.

frontend ft_web
  bind 0.0.0.0:8080
  # Stick table definition  
  stick-table type ip size 1m expire 15m store gpt0,http_req_rate(10s)
  http-request allow if { src -f /etc/hapee/whitelist.lst }
  http-request track-sc0 src
  acl abuse src_http_req_rate ge 5
  http-request sc-set-gpt0(0) 1 if abuse 
  http-request deny if { sc0_get_gpt0 ge 1 }

As mentioned, the above stick table can store up to 1 million entries, and each entry will contain 3 values: an IP address, the determined average HTTP request rate over 10 seconds, and a general purpose tag “GPT0”, which will have a value of “1” if the requests are to be denied. Entries in the table will be removed to not occupy space if the client IP makes no further requests within the configured expiration period (in our example, 15 minutes). Requests coming from the whitelisted IPs will be passed through without imposing any restrictions on them.

All of this will work automatically and require no manual intervention, thanks just to the configuration lines above.

However, we might still want to be able to manipulate the stick table during runtime, for example to change the entries’ GPT0 flag, to remove entries altogether, or to list currently blocked IPs. This can again be done using the Runtime API:

# List the contents of the stick table
> show table ft_web
# table: ft_web, type: ip, size:1048576, used:2
0x1c1efe0: key=127.0.0.1 use=0 exp=5600 gpt0=1 http_req_rate(10000)=50
0x1c0d8e0: key=192.168.122.1 use=0 exp=7713 gpt0=0 http_req_rate(10000)=1


# Set an entry's GPT0 value to 0
> set table ft_web key 127.0.0.1 data.gpt0 0


# Delete the entry
> clear table ft_web key 127.0.0.1

# Verify successful deletion
> show table ft_web
# table: ft_web, type: ip, size:1048576, used:2
0x1c0d8e0: key=192.168.122.1 use=0 exp=7700 gpt0=0 http_req_rate(10000)=0


# List remaining blocked client IPs (those with tag GPT0 value greater than 0)
> show table ft_web data.gpt0 gt 0

Updating ACLs

To extend the above example of rate-limiting the incoming HTTP requests, let’s assume we would now want to change the existing rate limit from 10 to 60 requests per second.

To do so, the HAProxy Runtime API will certainly allow us to change the ACLs during runtime. We will execute the command “show acl” to see the configured ACLs, then identify the one we want to modify, and finally perform the actual modification. The modification will consist of adding a new value and then deleting the old one to complete the transition.

A complete session could look like the following:

> show acl
# id (file) description               
0 () acl 'path_beg' file '/etc/hapee/hapee-lb.cfg' line 60
1 () acl 'nbsrv' file '/etc/hapee/hapee-lb.cfg' line 61
2 (/etc/hapee/whitelist.lst) pattern loaded from file '/etc/hapee/whitelist.lst' used by acl at file '/etc/hapee/hapee-lb.cfg' line 67
3 () acl 'src' file '/etc/hapee/hapee-lb.cfg' line 67
4 () acl 'src_http_req_rate' file '/etc/hapee/hapee-lb.cfg' line 69
5 () acl 'sc0_get_gpt0' file '/etc/hapee/hapee-lb.cfg' line 71
6 () acl 'path_beg' file '/etc/hapee/hapee-lb.cfg' line 80
7 () acl 'always_true' file '/etc/hapee/hapee-lb.cfg' line 106

# We are interested in the 5th ACL (ID #4)

# Display the ACL's current settings
> show acl #4
0xc9a3e0 10


# Add a new rate of 60
> add acl #4 60


# Verify that both values are now present
> show acl #4
0xc9a3e0 10
0xce8f10 60


# Delete the old value in any of the following two ways - by value, or by using the reference number
> del acl #4 10
> del acl #4 #0xc9a3e0


# Verify that only one value now remains in effect
> show acl #4
0xce8f10 60                         

Updating Whitelists and Blacklists

Again extending our original rate limiting example, please notice that we have used the following configuration directive to provision for whitelisted IP addresses:

http-request allow if { src -f /etc/hapee/whitelist.lst }

This configuration line tells HAProxy to fetch the source IP, check it against the list of IP addresses that were loaded from whitelist.lst, and pass the request through without further checks if the IP address is found in the list.

In this example we are going to modify the contents of the whitelist by using the Runtime API – we will be changing an IP address entry from “192.168.1.6” to “192.168.1.9”. As usual, to do so we are going to list the ACLs, identify the entries we want to modify, and then perform the actual modification. The modification will consist of adding the new value and then deleting the old one to complete the transition.

A complete session could look like the following:

> show acl

# id (file) description               
0 () acl 'path_beg' file '/etc/hapee/hapee-lb.cfg' line 60
1 () acl 'nbsrv' file '/etc/hapee/hapee-lb.cfg' line 61
2 (/etc/hapee/whitelist.lst) pattern loaded from file '/etc/hapee/whitelist.lst' used by acl at file '/etc/hapee/hapee-lb.cfg' line 67
3 () acl 'src' file '/etc/hapee/hapee-lb.cfg' line 67
4 () acl 'src_http_req_rate' file '/etc/hapee/hapee-lb.cfg' line 69
5 () acl 'sc0_get_gpt0' file '/etc/hapee/hapee-lb.cfg' line 71
6 () acl 'path_beg' file '/etc/hapee/hapee-lb.cfg' line 80
7 () acl 'always_true' file '/etc/hapee/hapee-lb.cfg' line 106

# We are interested in the third entry (ID #2)

# Display its content
> show acl #2

0x24de920 191.168.1.1                             
0x24de960 191.168.1.2
0x24de9a0 191.168.1.3
0x24de9e0 191.168.1.4
0x24dcf10 191.168.1.5
0x24de300 191.168.1.6
0x24de340 191.168.1.7


# Add the new entry
> add acl #2 192.168.1.9

# Delete the old entry
> del acl #2 #0x24de300

# Verify the result
> show acl #2
0x24de920 191.168.1.1                  
0x24de960 191.168.1.2
0x24de9a0 191.168.1.3
0x24de9e0 191.168.1.4
0x24dcf10 191.168.1.5
0x24de340 191.168.1.7
0x24de300 192.168.1.9

SSL/TLS

Updating TLS Tickets

HAProxy uses tls-ticket-keys to avoid the expensive key renegotiation when an existing client wants to start a new session after closing the previous one. If a client supports session tickets, HAProxy will send it a new session ticket record containing all of the negotiated session data (cipher suite, master secret, etc.). The record will be encrypted with a secret key only known to the HAProxy process, ensuring that the data won’t be read or modified by the client.

The secrets used to encrypt the TLS tickets are generated on HAProxy startup. If an active-active HAProxy cluster is set up and a client moves between nodes, it will have to renegotiate the keys often because the tickets, encrypted with node-specific keys, will not be understood by other nodes in the cluster. In order to avoid this, it is possible to place secrets in a file and also periodically rotate them to maintain perfect forward secrecy. That way all nodes in the cluster will use exactly the same set of secrets and clients will be able to move between HAProxy nodes without any side effects.

This could be done by updating the file containing the keys and reloading HAProxy manually, but it can also be done during runtime by using the HAProxy Runtime API.

A complete session could look like the following:

# List keys files
> show tls-keys
# id (file)               
0 (/dev/shm/hapee_ticket_keys.txt)


# List keys in a particular file
> show tls-keys #0
# id secret               
# 0 (/dev/shm/hapee_ticket_keys.txt)
0.0 vaL65b3ns0mAJbiQRGiQZ84H4C9N1dZAyDYrHXVqG6KRDuXCo8mpwfk6+xPtlM1m
0.1 fQxhSJT8sBKNb6JAFZT11UkzplfXEI1uUijPQUTBysZpNqzT26s2RVARxCoo5E52
0.2 QtxrjBbPrX6z/PljdHIFqmHMH2/Rc5zZzIKklcfBPJa01G6PU9Dp9ixcibeisZxU


# Update key 
> set ssl tls-keys #0 CaF7HpWr0gUByzxDqlbvYXCFT2zqmhnKFAdbM4MyQHfty974QO31Pn1OLJIR92rk
TLS ticket key updated!


# Verify successful update
> show tls-keys #0
                                                                  
# id secret
# 0 (/dev/shm/hapee_ticket_keys.txt)
0.0 fQxhSJT8sBKNb6JAFZT11UkzplfXEI1uUijPQUTBysZpNqzT26s2RVARxCoo5E52
0.1 QtxrjBbPrX6z/PljdHIFqmHMH2/Rc5zZzIKklcfBPJa01G6PU9Dp9ixcibeisZxU
0.2 CaF7HpWr0gUByzxDqlbvYXCFT2zqmhnKFAdbM4MyQHfty974QO31Pn1OLJIR92rk

Configuring OCSP Responses

HAProxy also supports the TLS Certificate Status Request extension, also known as “OCSP stapling”. For more information on configuring the OCSP responses, please see the documentation section 5.1 – crt. OCSP stapling allows HAProxy to send certificate status information to the clients. The information is a signed OCSP response and will avoid the need for a client to send a separate request to the OCSP service hosted by the issuing Certificate Authority. Here is an example of a command used to display the OCSP response from a server:

$ openssl s_client -servername hapee-mh -connect 127.0.0.1:443 -tlsextdebug -status
CONNECTED(00000003)                                     
TLS server extension "server name" (id=0), len=0
TLS server extension "renegotiation info" (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension "EC point formats" (id=11), len=4
0000 - 03 00 01 02                                       ....
TLS server extension "session ticket" (id=35), len=0
TLS server extension "status request" (id=5), len=0
TLS server extension "heartbeat" (id=15), len=1
0000 - 01                                                .
depth=2 C = FR, ST = France, O = MMH-LAB, OU = SERVER, CN = MMH-CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
OCSP response: 
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = FR, ST = France, O = MMH-LAB, OU = SERVER, CN = MMH-OCSP
    Produced At: Nov 20 16:00:00 2017 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 1A7A927EC43135F4B6FF62065A09E1C7FD02557B
      Issuer Key Hash: A45FE1C0DDC03B42C34306A08DF14ED33B74B86C
      Serial Number: 1000
    Cert Status: good
    This Update: Nov 20 16:00:00 2017 GMT
    Next Update: Nov 30 16:00:00 2017 GMT

Near the end of the output, we can see the information “Cert Status: Good”. We can also see in the last line of output that the next update is expected in 10 days.

Now, if we would want to update the OCSP response in order to change the next update time from 10 to 20 days, we could prepare the response and then load it into the running HAProxy instance by using the Runtime API:

$ openssl ocsp -CAfile intermediate/certs/ca-chain.cert.pem -issuer intermediate/certs/intermediate.cert.pem -cert intermediate/certs/hapee-mmh.cert.pem -url http://localhost:8080 -respout  /tmp/ocsp-resp.der  
                                                                                                
Response verify OK
intermediate/certs/hapee-mmh.cert.pem: good
        This Update: Nov 20 16:00:00 2017 GMT
        Next Update: Dec 10 16:00:00 2017 GMT

$ echo "set ssl ocsp-response $(base64 -w 10000 /tmp/ocsp-resp.der)" | socat stdio /var/run/hapee-lb.sock
OCSP Response updated!

And using the same approach as above, we can verify that HAProxy is now serving the updated response:

$ openssl s_client -servername hapee-mh -connect 127.0.0.1:443 -tlsextdebug -status

CONNECTED(00000003)
 ...
 ...
OCSP response: 
======================================
 ...
 ...
    Cert Status: good
    This Update: Nov 20 16:00:00 2017 GMT
    Next Update: Dec 10 16:00:00 2017 GMT

Troubleshooting

When it comes to troubleshooting the configuration or observed application behavior, looking into the HAProxy log files is probably one of the most common courses of action. And it is certainly a good one – HAProxy logs contain all sorts of information, including data about timers, sizes, connection counters, etc. But from time to time we may come across complicated issues or even bugs that need more detailed debugging output than even logs can provide.

Displaying Errors

As mentioned, HAProxy logs errors to log files, as documented in the section 1.3.1. The Response line. For example, if we send it simple, invalid traffic such as the following:

$ echo "This is invalid traffic\n\nMore invalid traffic" | nc -v hapee-lab 80
Connection to hapee-lab 80 port [tcp/http] succeeded!
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

400 Bad Request
Your browser sent an invalid request.

Then the logs will show the following error:

 hapee-lab hapee-lb[17817]: 192.168.122.1:53918 [20/Nov/2017:13:29:11.305] fe_main fe_main/ -1/-1/-1/-1/+0 400 +187 - - PR-- 0/0/0/0/0 0/0  ""

But to get even more information about the request in question and even see the contents of the request, we can use the Runtime API command “show errors”:

> show errors
Total events captured on [10/Nov/2017:13:41:58.421] : 3
 
[10/Nov/2017:13:41:43.328] frontend fe_main (#2): invalid request
  backend  (#-1), server  (#-1), event #2
  src 192.168.122.1:54154, session #15, session flags 0x00000080
  HTTP msg state MSG_RQVER(6), msg flags 0x00000000, tx flags 0x00000000
  HTTP chunk len 0 bytes, HTTP body len 0 bytes
  buffer flags 0x00808002, out 0 bytes, total 46 bytes
  pending 46 bytes, wrapping at 16384, error at position 8:
 
  00000  This is invalid traffic\n
  00024  \n
  00025  More invalid traffic\n

Dumping Sessions

Depending on the logging configuration, all connections can be logged to HAProxy’s log files. But, as usual, they are logged only after HAProxy gets a reply from the backend servers or when one of its timers expires. In situations where long timeouts are involved or where sessions are taking long to complete, this might cause the logs to never seem to arrive. In such situations, the Runtime API command “show sess” may be used to dump all current sessions and their related information:

> show sess
0xd38210: proto=tcpv4 src=192.168.122.1:55234 fe=fe_main be= srv= ts=02 age=11s calls=1rq[f=400000h,i=0,an=34h,rx=,wx=,ax=] rp[f=80000000h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[0,10h,fd=-1,ex=] exp=
0xd24dd0: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts=02 age=0s calls=1 rq[f=c08202h,i=0,an=00h,rx=10m,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[7,8h,fd=2,ex=] s1=[7,4018h,fd=-1,ex=] exp=10m

In the above example, we can see two sessions: the first one is an IPv4 session; the second is a session related to our invoking of the Runtime API. From looking into the output, we can, for example, identify that the IPv4 session is in an early stage of processing because no backend (“be=”) was selected yet. Also, to help us further in troubleshooting complex issues, we may use the command “show sess” with a session ID provided as an argument (“show sess ID”) to get even more details about a particular session.

Closing Sessions

In addition to displaying active sessions using “show sess”, we can also use the Runtime API to close sessions at will by using “shutdown session”:

> shutdown session 0xd38210

Such sessions will appear in the logs containing the flag “K”, indicating they were shut down.

Conclusion

We hope you have enjoyed this blog post providing an introduction to the HAProxy Runtime API and showing some of its most common and practical use cases. The complete HAProxy Runtime API documentation can be found in the HAProxy Management Guide, section 9.3.

If you have a subscription to HAProxy Enterprise Edition, we can provide you with authoritative support, scripts, and modules that will help you make the best use of the Runtime API, including its most advanced features listed among the commands but not specifically elaborated in this blog post.

The Runtime API is evolving along with all other features and improvements that we are adding to HAProxy. One of the planned Runtime API improvements is an HTTP REST interface to complement the existing access methods. Let us know what other features and improvements you would like to see included!