Security

Traffic policing

On this page

Traffic policing allows you to limit the rate and number of requests flowing to your backend servers. Traffic policing measures can ensure that users get the desired quality of service, and they can even prevent malicious traffic such as DDoS attacks.

In practice, traffic policing involves denying requests when request rates or counts exceed specified thresholds.

Queue connections to servers Jump to heading

It’s possible to use connection queueing to achieve the desired level of fairness without resorting to rate limiting. With connection queueing, the proxy stores excess connections until the servers are freed up to handle them. The load balancer is designed to hold lots of connections without a sharp increase in memory or CPU usage.

Queueing is disabled by default. To enable it:

  1. Add the maxconn argument to server directives. Use the maxconn argument to specify the maximum number of concurrent connections that will be established with the server.

    In the following example, up to 30 connections will be established to each server. Once all servers reach their maximum number of connections, new connections queue up in the load balancer:

    haproxy
    backend servers
      server s1 192.168.30.10:80 check  maxconn 30
      server s2 192.168.31.10:80 check  maxconn 30
      server s3 192.168.31.10:80 check  maxconn 30
    haproxy
    backend servers
      server s1 192.168.30.10:80 check  maxconn 30
      server s2 192.168.31.10:80 check  maxconn 30
      server s3 192.168.31.10:80 check  maxconn 30

    With this configuration, at most 90 connections can be active at a time. New connections will be queued on the proxy until an active connection closes.

  2. To define how long clients can remain in the queue, add the timeout queue directive:

    haproxy
    backend servers
      timeout queue 10s
      server s1 192.168.30.10:80 check  maxconn 30
      server s2 192.168.31.10:80 check  maxconn 30
      server s3 192.168.31.10:80 check  maxconn 30
    haproxy
    backend servers
      timeout queue 10s
      server s1 192.168.30.10:80 check  maxconn 30
      server s2 192.168.31.10:80 check  maxconn 30
      server s3 192.168.31.10:80 check  maxconn 30

    If a connection request still cannot be dispatched within the timeout period, the client receives a 503 Service Unavailable error. This error response is generally more desirable than allowing servers to become overwhelmed. From the client’s perspective, it’s better to receive a timely error that can be handled programmatically than to wait an extended amount of time and possibly cause errors that are more difficult to resolve.

Rate limit by IP address Jump to heading

To rate limit HTTP requests where you track clients by their IP addresses, define a stick table that has a type of ipv6, which can store IPv4 and IPv6 addresses. Use the fetch method sc_http_req_rate(<stick counter>) to fetch the current client’s rate. In the following example, we deny clients that make more than 100 requests in 10 seconds. The client receives a 429 Too Many Requests response. The time period of 10 seconds ends with the current request and is a sliding window. As requests age, they move out of the window, and the client is able to again make requests.

haproxy
peers mypeers
  peer local 127.0.0.1:10000
  table client_request_rates type ipv6  size 1m  expire 10s  store http_req_rate(10s)
frontend website
  bind :80
  http-request track-sc0 src table mypeers/client_request_rates
  acl exceeds_limit sc_http_req_rate(0,mypeers/client_request_rates) gt 100
  http-request deny deny_status 429 if exceeds_limit
  default_backend servers
haproxy
peers mypeers
  peer local 127.0.0.1:10000
  table client_request_rates type ipv6  size 1m  expire 10s  store http_req_rate(10s)
frontend website
  bind :80
  http-request track-sc0 src table mypeers/client_request_rates
  acl exceeds_limit sc_http_req_rate(0,mypeers/client_request_rates) gt 100
  http-request deny deny_status 429 if exceeds_limit
  default_backend servers

Depending on the period — here it’s 10 seconds — and how frequently the client makes requests, they may continue to exceed the rate limit for the current sliding window period. They might be blocked indefinitely if they don’t slow down for long enough. There’s another way to restrict clients that avoids accidentally denying them for a long time: Once the client exceeds the rate limit, flag them and deny them for a specific period of time. This works best when the deny period is longer than the sliding window of the rate limit period, giving the client time to back off, but without blocking them indefinitely.

Let’s see an example of that. In the next configuration snippet, we define two stick tables. One tracks each client’s request rate, while the other tracks whether they’ve been flagged as exceeding the limit. Notice that once a client has been flagged, we deny them. But to avoid denying them indefinitely, the http-request deny directive comes before the http-request track-sc0 and http-request track-sc1 directives, which short-circuits the processing and stops tracking the client’s requests until the record in the flagged_clients stick table expires and is removed. The converters that begin with table_, such as table_http_req_rate and table_gpt, can fetch a record from a stick table even when we skip tracking the current request, since you feed the source IP in via the src fetch.

Clients are flagged if they make more than 100 requests in 10 seconds. They’re denied then for 30 seconds, which is the value of the expire argument on the flagged_clients stick table. We return an HTTP header named Retry-After that tells the client how long in seconds until they can try again, based on the time until their stick table record expires. For that, use the table_expire converter. It returns milliseconds, so divide by 1,000.

haproxy
peers mypeers
  peer local 127.0.0.1:10000
  table client_request_rates type ipv6  size 1m  expire 10s  store http_req_rate(10s) 
  table flagged_clients type ipv6 size 1m expire 30s store gpt(1)
frontend website
  bind :80
  acl exceeds_limit src,table_http_req_rate(mypeers/client_request_rates) gt 100
  acl is_flagged src,table_gpt(0,mypeers/flagged_clients) eq 1
  http-request deny deny_status 429 if is_flagged
  http-after-response set-header Retry-After %[src,table_expire(mypeers/flagged_clients),div(1000)] if is_flagged
  http-request track-sc0 src table mypeers/client_request_rates
  http-request track-sc1 src table mypeers/flagged_clients
  http-request sc-set-gpt(0,1) 1 if exceeds_limit
  default_backend servers
haproxy
peers mypeers
  peer local 127.0.0.1:10000
  table client_request_rates type ipv6  size 1m  expire 10s  store http_req_rate(10s) 
  table flagged_clients type ipv6 size 1m expire 30s store gpt(1)
frontend website
  bind :80
  acl exceeds_limit src,table_http_req_rate(mypeers/client_request_rates) gt 100
  acl is_flagged src,table_gpt(0,mypeers/flagged_clients) eq 1
  http-request deny deny_status 429 if is_flagged
  http-after-response set-header Retry-After %[src,table_expire(mypeers/flagged_clients),div(1000)] if is_flagged
  http-request track-sc0 src table mypeers/client_request_rates
  http-request track-sc1 src table mypeers/flagged_clients
  http-request sc-set-gpt(0,1) 1 if exceeds_limit
  default_backend servers

Rate limit by URL path Jump to heading

You can assign distinct rate limits to individual URLs of your web application. This type of configuration can be useful when different pages require different amounts of processing time, and thus can handle a different number of concurrent users. This configuration uses a map file to associate different rate limits to different URLs in your web application.

  1. On the load balancer, create a file called rates.map.

  2. In the file, list the URL paths and rate thresholds. For example:

    haproxy
    /urla  10
    /urlb  20
    /urlc  30
    haproxy
    /urla  10
    /urlb  20
    /urlc  30

  3. Update the frontend configuration to include the stick-table and http-request track directives shown below:

    haproxy
    frontend website
      bind :80
      stick-table  type binary  len 20  size 100k  expire 10s  store http_req_rate(10s)
      http-request track-sc0 base32+src
      http-request set-var(req.rate_limit)  path,map_beg(/rates.map,20)
      http-request set-var(req.request_rate)  base32+src,table_http_req_rate()
      acl exceeds_limit var(req.rate_limit),sub(req.request_rate) lt 0
      http-request deny deny_status 429 if exceeds_limit
      default_backend servers
    haproxy
    frontend website
      bind :80
      stick-table  type binary  len 20  size 100k  expire 10s  store http_req_rate(10s)
      http-request track-sc0 base32+src
      http-request set-var(req.rate_limit)  path,map_beg(/rates.map,20)
      http-request set-var(req.request_rate)  base32+src,table_http_req_rate()
      acl exceeds_limit var(req.rate_limit),sub(req.request_rate) lt 0
      http-request deny deny_status 429 if exceeds_limit
      default_backend servers

    In this example:

    • The stick table has a key of binary to match the tracked value generated by the http-request track-sc0 base32+src directive, which is a hash of the HTTP Host header, the URL path, and the client’s source IP address. This key allows the load balancer to differentiate request rates across all different web pages.
    • The http-request set-var(req.rate_limit) directive retrieves the rate limit threshold from the rates.map file. This directive finds the request rate threshold in the rates.map file for the current URL path being requested. If the URL isn’t in the map file, a default value of 20 is used. The resulting threshold value is stored in the variable req.rate_limit.
    • The http-request set-var(req.request_rate) directive records the client’s request rate.
    • The ACL named exceeds_limit is set to true if the client’s request rate is greater than the rate limit threshold.
    • If the threshold is exceeded, the http-request deny directive denies the request.

Rate limit by URL parameter Jump to heading

As an alternative to rate limiting by URL path, you can configure request rate limiting by URL parameter. This approach can be useful if your clients include an API token in the URL to identify themselves. This configuration is based on a sliding window rate limit configuration.

In the following example, the client is expected to include a token with their requests, as follows:

text
http://yourwebsite.com/api/v1/does_a_thing?token=abcd1234
text
http://yourwebsite.com/api/v1/does_a_thing?token=abcd1234

For this example, the configuration applies a limit of 1,000 requests per 24 hour period, and it also requires that the user supply a token as shown above.

  1. In the frontend, add a stick table with a type of string to store the HTTP request rate. The sliding window size in this example is 24 hours:

    haproxy
    frontend website
      bind :80
      stick-table type string size 100k expire 24h store http_req_rate(24h)
      acl has_token url_param(token) -m found
      acl exceeds_limit url_param(token),table_http_req_rate() gt 1000
      http-request track-sc0 url_param(token) unless exceeds_limit
      http-request deny deny_status 429 if !has_token or exceeds_limit
    haproxy
    frontend website
      bind :80
      stick-table type string size 100k expire 24h store http_req_rate(24h)
      acl has_token url_param(token) -m found
      acl exceeds_limit url_param(token),table_http_req_rate() gt 1000
      http-request track-sc0 url_param(token) unless exceeds_limit
      http-request deny deny_status 429 if !has_token or exceeds_limit

    In this example:

    • The ACL named has_token indicates if the desired token is included in the URL.
    • The ACL named exceeds_limit finds the current request count for the last 24 hours and compares it to the request rate limit threshold, 1000.
    • The http-request track directive stores the value of the URL parameter named token as the key in the table. The unless exceeds_limit clause serves an important purpose. It prevents the counter from continuing to increment once the client has exceeded the limit. The clause also allows the entry to expire so that the client isn’t permanently blocked.
    • The http-request deny directive denies the request if the token is missing or if the limit is exceeded.

There is an important reason why this configuration uses the http_req_rate(24h) counter instead of the http_req_cnt counter in conjunction with an expire parameter set to 24h. The former is a sliding window over the last 24 hours. The latter begins when the user sends their first request and increments from then on until the expiration. However, unless you’re manually clearing the table every 24 hours via the Runtime API, the http_req_cnt could stay in effect for a long time while the client stays active. That’s because the expiration is reset whenever the record is touched.

Limit requests per day (sliding window) Jump to heading

You can limit the number of HTTP requests a user can make within a period of time, such as 24 hours. The period of time ends with the current request and includes requests that happened during the preceding period. This kind of limit is called a sliding window rate limit. It works the same way as rate limiting by IP address, but with a longer period.

In this example, the sliding window limit allows each client to issue no more than 1000 requests during a 24-hour period:

haproxy
frontend website
  bind :80
  stick-table  type ipv6  size 1m  expire 24h  store http_req_rate(24h)
  http-request track-sc0 src
  http-request deny deny_status 429 if { sc_http_req_rate(0) gt 1000 }
  default_backend servers
haproxy
frontend website
  bind :80
  stick-table  type ipv6  size 1m  expire 24h  store http_req_rate(24h)
  http-request track-sc0 src
  http-request deny deny_status 429 if { sc_http_req_rate(0) gt 1000 }
  default_backend servers

  • With the stick-table directive, you define a table that stores each client’s HTTP request rate over the given period, where each client is tracked by IP address. To conserve space, the stick table size is set to the 1m most recently active clients, where 1m means 1,048,576. Also, entries expire and are removed if they are inactive for 24 hours, ensuring we keep a record for the same amount of time as the sliding window request rate period.

  • The http-request track directive stores the client’s IP address with their request rate in the stick table.

  • The http-request deny directive denies requests for clients that exceed the limit. On the http-request deny line, the if expression determines whether the client’s current request rate has exceeded the allowed number of requests, in this case 1000. If so, we deny the current request with a 429 Too Many Requests response. When the count of requests during the preceding 24 hours is again below 1000, we accept the request.

You can adjust any part of this example to suit your needs.

  • To change the time period, change the time specified in the http_req_rate fetch in the stick-table directive. For a one-minute period, use 1m:

    haproxy
    stick-table  type ipv6  size 1m  expire 1m  store http_req_rate(1m)
    haproxy
    stick-table  type ipv6  size 1m  expire 1m  store http_req_rate(1m)

  • To change the number of allowable requests in the interval, change the gt test value specified in the http-request deny directive. For 10,000 requests, set it like this:

    haproxy
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 10000 }
    haproxy
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 10000 }

Limit requests per day (fixed window) Jump to heading

Limitations

This technique requires the HAProxy Runtime API, which isn’t available with HAProxy ALOHA.

A fixed window request limit restricts the number of requests that a client can send during a fixed period of time, such as a calendar day. In this example, we configure a limit of 1,000 HTTP requests during a calendar day. The http_req_cnt counter stores the running total of requests each client makes, based on IP address, as a 32-bit integer.

haproxy
frontend website
  bind :80
  stick-table  type ipv6  size 100k  expire 24h  store http_req_cnt
  http-request track-sc0 src
  http-request deny deny_status 429 if { sc_http_req_cnt(0) gt 1000 }
  default_backend servers
haproxy
frontend website
  bind :80
  stick-table  type ipv6  size 100k  expire 24h  store http_req_cnt
  http-request track-sc0 src
  http-request deny deny_status 429 if { sc_http_req_cnt(0) gt 1000 }
  default_backend servers

  • In the frontend, the stick-table directive defines the data we want to store. It has a type of ipv6 to store IPv4 and IPv6 client IP addresses.

  • The http-request track directive stores the client’s IP address with their request count in the stick table. With each new request, the count increments.

  • The http-request deny directive denies requests for clients that exceed the limit.

This configuration causes every request after 1,000 to be denied, but we need to reset that restriction at midnight. To reset the counter, use the Runtime API’s clear table command:

  1. Enable the Runtime API.

  2. Install the socat utility, and use it to invoke the clear table Runtime API command to clear all records from the stick table:

    nix
    echo "clear table website" |\
      sudo socat stdio unix-connect:/var/run/hapee-3.3/hapee-lb.sock
    nix
    echo "clear table website" |\
      sudo socat stdio unix-connect:/var/run/hapee-3.3/hapee-lb.sock

  3. To reset the counter automatically, set up a daily cron job.

Other response policies Jump to heading

So far, our examples have used http-request deny to drop HTTP requests that exceed the rate limit. However, there are other policies you can implement.

Deny an HTTP request Jump to heading

You can deny a client’s HTTP request to return a 403 Forbidden error to the client. Use the directive http-request deny in your frontend section. In the example below, we deny the client’s request if they’ve made more than 20 requests within the last minute:

haproxy
frontend website
  bind :80
  # use a stick table to track request rates
  stick-table type ip size 100k expire 1m store http_req_rate(1m)
  http-request track-sc0 src
  acl too_many_requests sc_http_req_rate(0) gt 20
  http-request deny if too_many_requests
  default_backend webservers
haproxy
frontend website
  bind :80
  # use a stick table to track request rates
  stick-table type ip size 100k expire 1m store http_req_rate(1m)
  http-request track-sc0 src
  acl too_many_requests sc_http_req_rate(0) gt 20
  http-request deny if too_many_requests
  default_backend webservers

You also change the response code by setting the deny_status argument. Below, we return a 429 Too Many Requests error:

haproxy
http-request deny deny_status 429 if too_many_requests
haproxy
http-request deny deny_status 429 if too_many_requests

Tarpit an HTTP request Jump to heading

You can tarpit a client’s HTTP request, which stalls the request for a period of time before returning an error response. This is often used to deter a malicious bot army, since it ties up bots so that they cannot immediately retry their requests.

In the example below, we use http-request tarpit to impede the client if they exceed a rate limit. Use timeout tarpit to set how long the load balancer waits before returning an error response:

haproxy
frontend www
  bind :80
  # use a stick table to track request rates
  stick-table type ip size 100k expire 1m store http_req_rate(1m)
  http-request track-sc0 src
  acl too_many_requests sc_http_req_rate(0) gt 20
  timeout tarpit 10s
  http-request tarpit deny_status 429 if too_many_requests
  default_backend webservers
haproxy
frontend www
  bind :80
  # use a stick table to track request rates
  stick-table type ip size 100k expire 1m store http_req_rate(1m)
  http-request track-sc0 src
  acl too_many_requests sc_http_req_rate(0) gt 20
  timeout tarpit 10s
  http-request tarpit deny_status 429 if too_many_requests
  default_backend webservers

Pause an HTTP request or response Jump to heading

This section applies to:

  • HAProxy 3.2 and newer
  • HAProxy Enterprise 3.2 and newer
  • HAProxy ALOHA 17.5 and newer

You can delay processing an HTTP request or response for a period of time using the pause argument with the http-request and http-response directives. The pause argument must be set with a timeout value (default is milliseconds) or an expression that returns a timeout value (where errors and invalid values are ignored).

Use cases include slowing down clients that exceed a rate limit or for debugging purposes.

haproxy
defaults example1
  http-request pause 20   # pause request for 20 milliseconds
  http-response pause 20  # pause response for 20 milliseconds
  ...
frontend example2
  http-request pause res.hdr(X-Pause-Seconds),mul(1000)  # pause request for x seconds
  http-response pause res.hdr(X-Pause-Seconds),mul(1000) # pause response for x seconds
  ...
backend example3
  http-request pause 20s  # pause request for 20 seconds with "s" suffix
  http-response pause 20s # pause response for 20 seconds with "s" suffix
  ...
haproxy
defaults example1
  http-request pause 20   # pause request for 20 milliseconds
  http-response pause 20  # pause response for 20 milliseconds
  ...
frontend example2
  http-request pause res.hdr(X-Pause-Seconds),mul(1000)  # pause request for x seconds
  http-response pause res.hdr(X-Pause-Seconds),mul(1000) # pause response for x seconds
  ...
backend example3
  http-request pause 20s  # pause request for 20 seconds with "s" suffix
  http-response pause 20s # pause response for 20 seconds with "s" suffix
  ...

Reject a TCP connection Jump to heading

You can reject a TCP connection, which closes the connection immediately without sending a response. The client’s browser will display the error message: The connection was reset. Use one of the following directives:

Directive Description
http-request reject Closes the connection without a response after a session has been created and the HTTP parser has been initialized. Use this if you need to evaluate the request’s Layer 7 attributes (HTTP headers, cookies, URL).
tcp-request content reject Closes the connection without a response once a session has been created, but before the HTTP parser has been initialized. These requests still show in your logs.
tcp-request connection reject Closes the connection without a response at the earliest point, before a session has been created. These requests don’t show in your logs.

In the following example, we reject connections originating from IP addresses we wish to block:

haproxy
frontend www
  bind :80
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  tcp-request connection reject if banned_ip
  default_backend webservers
haproxy
frontend www
  bind :80
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  tcp-request connection reject if banned_ip
  default_backend webservers

Silently replace the URL path Jump to heading

You can silently replace the URL path of an incoming client request before sending it to a backend server. The client is unaware of the replacement, and their browser address bar doesn’t update to reflect it. Use cases include replacing cookies, consolidating address schemes, stripping URL prefixes, or hiding directory structures.

In the following example, the /example-test/app/survey cookie will be removed from the URL path:

haproxy
http-request replace-path (.*) /example-test/app/survey
haproxy
http-request replace-path (.*) /example-test/app/survey

See the configuration manual’s replace-path reference for further details.

Silently drop a TCP connection Jump to heading

You can silently drop a client’s TCP connection, which disconnects immediately without notifying the client that the connection has been closed. This means that the load balancer frees any resources used for this connection. Clients will typically need to time out before they can release their end of the connection. Beware that silently dropping will affect any stateful firewalls or proxies between the load balancer and the client since they will hold onto the connection, unaware that it has been disconnected.

In the example below, we use http-request silent-drop to silently drop clients that access a restricted file:

haproxy
frontend www
  bind :80
  http-request silent-drop if { path_end /restricted.txt }
  default_backend webservers
haproxy
frontend www
  bind :80
  http-request silent-drop if { path_end /restricted.txt }
  default_backend webservers

Reject a QUIC connection Jump to heading

This section applies to:

  • HAProxy 3.1 and newer
  • HAProxy Enterprise 3.1 and newer
  • HAProxy ALOHA 17.0 and newer

You can reject a QUIC connection, which closes the connection before the TLS handshake and sends a CONNECTION_REFUSED error code to the client. After failing to connect via HTTP/3 over QUIC, the client (browser) will typically fall back to using HTTP/2 over TCP. To block the client completely, you’ll also need to add rules that reject the TCP connection.

Below, we use quic-initial reject to reject connections originating from IP addresses we wish to block:

haproxy
frontend www
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  bind quic4@:443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  http-response set-header alt-svc 'h3=":443"; ma=900'
  http-request redirect scheme https unless { ssl_fc }
  # Reject QUIC packets from banned IPs
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  quic-initial reject if banned_ip
  default_backend webservers
haproxy
frontend www
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  bind quic4@:443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  http-response set-header alt-svc 'h3=":443"; ma=900'
  http-request redirect scheme https unless { ssl_fc }
  # Reject QUIC packets from banned IPs
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  quic-initial reject if banned_ip
  default_backend webservers

Silently ignore a QUIC connection Jump to heading

This section applies to:

  • HAProxy 3.1 and newer
  • HAProxy Enterprise 3.1 and newer
  • HAProxy ALOHA 17.0 and newer

You can silently ignore a QUIC connection by using quic-initial dgram-drop. This ignores the reception of a QUIC Initial packet, preventing a QUIC connection in the first place.

haproxy
frontend www
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  bind quic4@:443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  http-response set-header alt-svc 'h3=":443"; ma=900'
  http-request redirect scheme https unless { ssl_fc }
  # Reject QUIC packets from banned IPs
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  quic-initial dgram-drop if banned_ip
  default_backend webservers
haproxy
frontend www
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  bind quic4@:443 ssl crt /etc/haproxy/certs/example.com.pem ssl-min-ver TLSv1.3
  http-response set-header alt-svc 'h3=":443"; ma=900'
  http-request redirect scheme https unless { ssl_fc }
  # Reject QUIC packets from banned IPs
  acl banned_ip src -f /etc/hapee-3.3/blocklist.acl
  quic-initial dgram-drop if banned_ip
  default_backend webservers

See also Jump to heading

  • Stick tables can be used for a variety of purposes. For more information, see Stick tables.
  • To clear a stick table without interrupting the load balancer, use the clear table Runtime API command.

For complete information on directives and actions used in traffic policing, see these topics in the HAProxy Configuration Manual:

Do you have any suggestions on how we can improve the content of this page?

Previous page OCSP stapling Next page Reliability