Global Profiling Engine

The Global Profiling Engine collects stick table data from all HAProxy Enterprise nodes in a cluster. It can perform calculations on both current and historical data before pushing the data back to the load balancers, giving them all a comprehensive and up-to-date view of client behavior across the cluster.

For example, in an active-active load balancer configuration where traffic is relayed to two or more load balancers in a round-robin rotation, the GPE ensures that each load balancer receives the sum of requests that a client has made, even when some of those requests were routed to the other load balancer.

It calculates statistics for current, real-time traffic, which you can use to take action based on an immediate threat, such as a web scraper. It also calculates historical statistics, which you can use to make decisions based on traffic trends from the past. For example, you can factor in the request rate at the 90th percentile from yesterday.

The Global Profiling Engine should be installed on a separate server from your HAProxy Enterprise nodes, from where it can collect and aggregate data from stick tables from across your load balancer cluster.

When would you use the profiling engine? The table below outlines various, common ways to set up your load-balancing cluster and explains which benefit from using the Global Profiling Engine:

Standalone load balancer

If you operate a single HAProxy Enterprise node, collecting data with stick tables can help improve security and observability by monitoring client activity in real time. However, you do not need to use the Global Profiling Engine because there aren’t multiple load balancers collecting data that needs to be aggregated, unless you want to use its historical aggregations feature.

Active-standby cluster

In an active-standby cluster, you operate two HAProxy Enterprise nodes as peers, with one peer actively receiving traffic while the other is on standby. If you define a peers section in your configuration, then the stick tables of the active node are propagated to the passive node, where they overwrite existing values. Overwriting would be bad if both peers were receiving traffic and collecting data, but in an active-passive scenario, it’s perfectly fine. The metrics stay accurate on all nodes. In this case, you do not need the Global Profiling Engine, which aggregates data from multiple, active peers, unless you want to use its historical aggregations feature.

Active-active cluster with Global Profiling Engine

In an active-active cluster, you operate two HAProxy Enterprise nodes as peers, with both receiving traffic at the same time. To ensure that stick table counters are accurate cluster-wide, you need the Global Profiling Engine, which runs on a separate server, collecting stick table data from both peers and aggregating it. Once aggregated, the data is sent back to the peers so that they can make decisions based on it.

Installation Jump to heading #

The following section gives detailed information on how to install the Global Profiling Engine.

1. Provision a server on which you will install the Global Profiling Engine.

2. On the new server, add package repositories for your operating system.

   debian ubuntu redhat

   In a terminal window, set a variable named key to your HAProxy Enterprise license key.

   nix
   key=<HAProxy Enterprise Key>

   nix
   key=<HAProxy Enterprise Key>

   Then use the echo and tee commands to add the HAProxy Technologies extras repository location into the file /etc/apt/sources.list.d/haproxy-tech.list, as shown below.

   nix
   echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/debian-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

   nix
   echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/debian-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

   In a terminal window, set a variable named key to your HAProxy Enterprise license key.

   nix
   key=<HAProxy Enterprise Key>

   nix
   key=<HAProxy Enterprise Key>

   Then use the echo and tee commands to add the HAProxy Technologies extras repository location into the file /etc/apt/sources.list.d/haproxy-tech.list, as shown below.

   nix
   echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/ubuntu-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

   nix
   echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/ubuntu-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

   Create a new file /etc/yum.repos.d/haproxy-tech.repo if it does not exist and add the contents below. Replace <HAProxy Enterprise Key> with the key you were given when you registered. Also replace rhel-8 with your RHEL version number.

   haproxy-tech.repo
   ini
   [hapee-plus-extras]
   name=hapee-plus-extras
   enabled=1
   baseurl=https://www.haproxy.com/download/hapee/key/<HAProxy Enterprise Key>-plus/extras/rhel-8/$basearch/bin/
   gpgcheck=1

   haproxy-tech.repo
   ini
   [hapee-plus-extras]
   name=hapee-plus-extras
   enabled=1
   baseurl=https://www.haproxy.com/download/hapee/key/<HAProxy Enterprise Key>-plus/extras/rhel-8/$basearch/bin/
   gpgcheck=1

3. Install packages:

   debian ubuntu redhat

   nix
   sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
   wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
   sudo apt-get update
   sudo apt-get install --yes hapee-extras-gpe

   nix
   sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
   wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
   sudo apt-get update
   sudo apt-get install --yes hapee-extras-gpe

   nix
   sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
   wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
   sudo apt-get update
   sudo apt-get install --yes hapee-extras-gpe

   nix
   sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
   wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
   sudo apt-get update
   sudo apt-get install --yes hapee-extras-gpe

   nix
   rpm --import https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc
   yum makecache
   yum install -y hapee-extras-gpe

   nix
   rpm --import https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc
   yum makecache
   yum install -y hapee-extras-gpe

4. To start the Global Profiling Engine, run:

   nix
   sudo systemctl enable hapee-extras-gpe
   sudo systemctl start hapee-extras-gpe

   nix
   sudo systemctl enable hapee-extras-gpe
   sudo systemctl start hapee-extras-gpe

Manage the service Jump to heading #

Start, stop and check the status of the Global Profiling Engine service by using the corresponding systemctl commands.

Start the service:

nix
sudo systemctl start hapee-extras-gpe

nix
sudo systemctl start hapee-extras-gpe

Restart the service:

nix
sudo systemctl restart hapee-extras-gpe

nix
sudo systemctl restart hapee-extras-gpe

Show current status:

nix
sudo systemctl status hapee-extras-gpe

nix
sudo systemctl status hapee-extras-gpe

Stop the service:

nix
sudo systemctl stop hapee-extras-gpe

nix
sudo systemctl stop hapee-extras-gpe

Real-time aggregation Jump to heading #

The Global Profiling Engine collects stick table data from all HAProxy Enterprise nodes in the cluster in real time. It then aggregates that data and pushes it back to all of the nodes. For example, if LoadBalancer1 receives two requests and LoadBalancer2 receives three requests, the Global Profiling Engine will sum those numbers to get a total of five, then push that to both LoadBalancer1 and LoadBalancer2. This is helpful for an active/active load balancer configuration wherein the nodes need to share client request information to form an accurate picture of activity across the cluster.

The aggregated data does not overwrite the data on the load balancer nodes. Instead, it is pushed to secondary stick tables that have, for example, a suffix of .aggregate. You would use a fetch method to retrieve the aggregated data and perform an action, such as rate limiting.

Stick table data is transferred between the HAProxy Enterprise servers and the Global Profiling Engine server by using the peers protocol, a protocol created specifically for this purpose. You must configure which servers should participate, both on the Global Profiling Engine server and on each HAProxy Enterprise node.

Configure HAProxy Enterprise nodes Jump to heading #

An HAProxy Enterprise node must be configured to share its stick table data with the Global Profiling Engine server. Once aggregated, the profiling engine sends the data back to each node where it is stored in a new stick table. Follow these steps on each load balancer:

1. Edit the file /etc/hapee-2.8/hapee-lb.cfg.

   Add a peers section.

   hapee-lb.cfg
   haproxy
   global
     # By setting this, you tell HAProxy Enterprise to use the server line
     # that specifies this name as the local node.
     localpeer enterprise1
   peers mypeers
     bind 0.0.0.0:10000
     
     # the local HAProxy Enterprise node
     # name should match the server's hostname or the 'localpeer' value
     server enterprise1
     
     # The Global Profiling Engine
     # If you run GPE on the same server, use a different port here
     server gpe 192.168.50.40:10000
     
     # stick tables definitions
     table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
     table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

   hapee-lb.cfg
   haproxy
   global
     # By setting this, you tell HAProxy Enterprise to use the server line
     # that specifies this name as the local node.
     localpeer enterprise1
   peers mypeers
     bind 0.0.0.0:10000
     
     # the local HAProxy Enterprise node
     # name should match the server's hostname or the 'localpeer' value
     server enterprise1
     
     # The Global Profiling Engine
     # If you run GPE on the same server, use a different port here
     server gpe 192.168.50.40:10000
     
     # stick tables definitions
     table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
     table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

   Inside it:

   • Define a bind line to set the IP address and port at which this node should receive data back from the Global Profiling Engine server. The server name value is important because it must match the name you set in the Global Profiling Engine server’s configuration for the corresponding peer line. In this example, the bind directive listens on all IP addresses at port 10000 and receives aggregated data.

   • Define a server line for the current server. Ensure the host name matches the name determined by one of the following methods, in order of precedence:

     • the -L argument specified in the command line used to start the load balancer process;
     • the localpeer name specified in the global section of the load balancer configuration; or
     • the host name returned by the system hostname command. Method 3 is the default, but the other methods are recommended instead.

     In this example, the local HAProxy Enterprise node is listed with only its hostname, enterprise1. It is not necessary to specify its IP address and port.

   • Define a server line for the Global Profiling Engine server. Set its IP address and port. The name you set here is also important. It must match the corresponding peer line in the Global Profiling Engine server’s configuration.

   • Define stick tables. For each one, add a duplicate line where the table name has the suffix .aggregate. In this example, the non-aggregated stick table request_rates will store current HTTP request rates. The stick tables record the rate at which clients make requests over 10 seconds. We clear out stale records after 30 seconds by setting the expire parameter on the stick table. The type parameter sets the key for the table, which in this case is an IP address. The stick table request_rates.aggregate receives its data from the Global Profiling Engine. Its suffix, .aggregate, will match the the profiling engine’s configuration.

2. Add directives to your frontend, backend or listen sections that populate the non-aggregated stick tables with data.

   Below, the http-request track-sc0 line adds request rate information for each client that connects to the load balancer, using the client’s source IP address (src) as the key in the stick table.

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
     default_backend webservers
     
     # add records to the stick table using the client's
     # source IP address as the table key
     http-request track-sc0 src table mypeers/request_rates

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
     default_backend webservers
     
     # add records to the stick table using the client's
     # source IP address as the table key
     http-request track-sc0 src table mypeers/request_rates

3. Add directives that read the aggregated data returned from the Global Profiling Engine server. That data is stored in the table with the suffix .aggregate.

   Below, the http-request deny line rejects clients that have a request rate greater than 1000. The client’s request rate is an aggregate amount calculated from all active load balancers. Note that this line reads data from the request_rates.aggregate table.

   hapee-lb.cfg
   haproxy
   # perform actions like rate limiting
   http-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 }

   hapee-lb.cfg
   haproxy
   # perform actions like rate limiting
   http-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 }

4. Restart HAProxy Enterprise.

   nix
   sudo systemctl restart hapee-2.8-lb

   nix
   sudo systemctl restart hapee-2.8-lb

Configure the Global Profiling Engine Jump to heading #

The Global Profiling Engine server collects stick table data from HAProxy Enterprise load balancers in your cluster, but you must set which load balancers will be allowed to participate by listing them in the configuration file.

1. On the Global Profiling Engine server, edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

   In the aggregations section, add a peer line for the Global Profiling Engine itself and for each HAProxy Enterprise node. Each peer’s name (e.g. enterprise1) should match the name you set in the HAProxy Enterprise configuration, since that is how the profiling engine validates the peer.

   hapee-gpe-stktagg.cfg
   haproxy
   global
     # Enables the Global Profiling Engine API
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations data
     # set how to map non-aggregated to aggregated stick tables
     from any to .aggregate
     
     # the profiling engine listens at this address
     peer gpe 0.0.0.0:10000 local
     
     # the load balancers listen at these addresses
     peer enterprise1 192.168.50.41:10000
     peer enterprise2 192.168.50.42:10000

   hapee-gpe-stktagg.cfg
   haproxy
   global
     # Enables the Global Profiling Engine API
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations data
     # set how to map non-aggregated to aggregated stick tables
     from any to .aggregate
     
     # the profiling engine listens at this address
     peer gpe 0.0.0.0:10000 local
     
     # the load balancers listen at these addresses
     peer enterprise1 192.168.50.41:10000
     peer enterprise2 192.168.50.42:10000

   In this example:

   • The Global Profiling Engine API provides a programmable API, which listens at the socket /var/run/hapee-extras/gpe-api.sock. The stats socket directive enables a CLI that lets you view data that the aggregator has stored.

   • In the aggregations section, the from line defines how non-aggregated stick tables map to aggregated stick tables, and what the suffix for the aggregated stick tables should be. The keyword any means that any stick table found will be aggregated. Aggregated data is pushed to tables with the same name, but ending with the suffix .aggregate. In the example, the engine expects stick tables to be named like request_rates and it will push aggregated data to request_rates.aggregate.

     You can also use a more specific mapping. In the example below, the engine expects stick tables to be named like request_rates.nonaggregate and it will push aggregated data to request_rates.aggregate. Stick tables without the .nonaggregate suffix will be ignored.

     haproxy
     from .nonaggregate to .aggregate

     haproxy
     from .nonaggregate to .aggregate

   • HAProxy Enterprise peer lines must use the same name you set on the server line in the HAProxy Enterprise configuration (e.g. enterprise1), and specify the IP address and port where that node is receiving aggregated data.

2. Restart the Global Profiling Engine service:

   haproxy
   sudo systemctl restart hapee-extras-gpe

   haproxy
   sudo systemctl restart hapee-extras-gpe

Verify your setup Jump to heading #

Check that the Global Profiling Engine and load balancers are setup correctly by utilizing their APIs.

1. On the load balancer, call the Runtime API function show peers to check that the Global Profiling Engine is listed and that its last_status is ESTA (established):

   Below, the show peers command lists connected peers:

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock | head -2

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock | head -2

   output
   text
   0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=92
   0x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17s

   output
   text
   0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=92
   0x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17s

2. Call the Runtime API function show table to see data in non-aggregated and aggregated stick tables.

   Below, we view data in the stick table named request_rates.aggregate:

   nix
   echo "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock

   nix
   echo "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock

   output
   text
   # table: mypeers/request_rates.aggregate, type: ip, size:102400, used:1
   0x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5

   output
   text
   # table: mypeers/request_rates.aggregate, type: ip, size:102400, used:1
   0x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5

3. On the Global Profiling Engine server, call the show aggrs function to see load balancers that are registered as peers. A state of 0x7 means a successful connection. If you see a state of 0xffffffff, that means that a connection was not successful. Often, this is caused by the peer names not matching between the Global Profiling Engine’s configuration and the HAProxy Enterprise configuration.

   Below, the show aggrs command shows that the peer named enterprise1 has connected:

   nix
   echo "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.sock

   nix
   echo "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.sock

   output
   text
   aggregations data
     peer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3
            sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0

   output
   text
   aggregations data
     peer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3
            sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0

Optional: Bind outgoing connections to an interface Jump to heading #

If the server where you are running the Global Profiling Engine has multiple network interfaces, you can configure the engine to bind to a specific one for outgoing data sent to HAProxy Enterprise servers.

To bind outgoing connections to a specific address, use the source directive in the global section.

IPv4 examples

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12:12345

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12:12345

The port is optional. It defaults to 0 for random ports.

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12

IPv6 examples

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]:12345

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]:12345

The port is optional. It defaults to 0 for random ports.

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]

Multi-level setup Jump to heading #

You can aggregate stick tables from other Global Profiling Engines, which allows you to aggregate stick tables across different data centers, for example.

We will consider the following setup:

The top-level aggr3 Global Profiling Engine will sum the counters from the intermediate aggr1 and aggr2 aggregate stick tables. It will then send the top-level aggregate stick table to all HAProxy Enterprise nodes.

You can also host multiple top-level servers for high availability. In that case, intermediate servers simply push their data to both. See below for details.

Configure the top-level Global Profiling Engine Jump to heading #

Follow these steps on the server you wish to be the top-level Global Profiling Engine.

1. Edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations toplevel
     from .intermediate to .aggregate
     peer top-profiling-engine  0.0.0.0:10000 local
     peer intermediate-profiling-engine1  192.168.56.111:10000 down
     peer intermediate-profiling-engine2  192.168.56.112:10000 down

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations toplevel
     from .intermediate to .aggregate
     peer top-profiling-engine  0.0.0.0:10000 local
     peer intermediate-profiling-engine1  192.168.56.111:10000 down
     peer intermediate-profiling-engine2  192.168.56.112:10000 down

   • The current server has the local keyword set on its peer line.

   • In this example, two other Global Profiling Engine servers, intermediate-profiling-engine1 and intermediate-profiling-engine2, are listed with the down keyword, which means that they are one level down from the top.

   • The top-level Global Profiling Engine will aggregate stick table data from the intermediate servers. Their stick tables should have the .intermediate suffix.

   • The top-level Global Profiling Engine will push aggregated data back to the intermediate servers. The globally aggregated stick tables should have the .aggregate suffix.

Configure the intermediate Global Profiling Engines Jump to heading #

Follow these steps on the servers you wish to be the intermediate-level Global Profiling Engines.

1. Edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

   intermediate-profiling-engine1

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations myaggr
     from any to .intermediate
     forward .aggregate
     peer intermediate-profiling-engine1  0.0.0.0:10000 local
     peer top-profiling-engine  192.168.56.113:10000 up
     peer enterprise1 192.168.50.41:10000
     peer enterprise2 192.168.50.42:10000

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations myaggr
     from any to .intermediate
     forward .aggregate
     peer intermediate-profiling-engine1  0.0.0.0:10000 local
     peer top-profiling-engine  192.168.56.113:10000 up
     peer enterprise1 192.168.50.41:10000
     peer enterprise2 192.168.50.42:10000

   intermediate-profiling-engine2

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations myaggr
     from any to .intermediate
     forward .aggregate
     peer intermediate-profiling-engine2  0.0.0.0:10000 local
     peer top-profiling-engine  192.168.56.113:10000 up
     peer enterprise3 192.168.50.51:10000
     peer enterprise4 192.168.50.52:10000

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations myaggr
     from any to .intermediate
     forward .aggregate
     peer intermediate-profiling-engine2  0.0.0.0:10000 local
     peer top-profiling-engine  192.168.56.113:10000 up
     peer enterprise3 192.168.50.51:10000
     peer enterprise4 192.168.50.52:10000

   • The current server has the local keyword set on its peer line.

   • The upper-level Global Profiling Engine peer is denoted by the up keyword.

   • Each intermediate Global Profiling Engine is aware of only the HAProxy Enterprise nodes it manages and of the top-level Global Profiling Engine.

   • The intermediate-level Global Profiling Engines will aggregate stick table data from the HAProxy Enterprise servers.

   • The forward line relays the top-level server’s .aggregate stick tables to the HAProxy Enterprise servers.

   • The intermediate-level Global Profiling Engines will push aggregated data back to the HAProxy Enterprise servers. The aggregated stick tables should have the .aggregate suffix.

Configure for high availability Jump to heading #

To create a highly available setup, you can have multiple top-level servers. Add the group parameter to them so that the intermediate servers recognize that there are two top-level servers.

intermediate-profiling-engine1

hapee-gpe-stktagg.cfg
haproxy
global
  stats socket /var/run/hapee-extras/gpe-api.sock
aggregations myaggr
  from any to .intermediate
  forward .aggregate
  peer intermediate-profiling-engine1  0.0.0.0:10000 local
  peer top-profiling-engine1  192.168.56.113:10000 up group 1
  peer top-profiling-engine2  192.168.56.114:10000 up group 1
  peer enterprise1 192.168.50.41:10000
  peer enterprise2 192.168.50.42:10000

hapee-gpe-stktagg.cfg
haproxy
global
  stats socket /var/run/hapee-extras/gpe-api.sock
aggregations myaggr
  from any to .intermediate
  forward .aggregate
  peer intermediate-profiling-engine1  0.0.0.0:10000 local
  peer top-profiling-engine1  192.168.56.113:10000 up group 1
  peer top-profiling-engine2  192.168.56.114:10000 up group 1
  peer enterprise1 192.168.50.41:10000
  peer enterprise2 192.168.50.42:10000

Historical aggregation Jump to heading #

In addition to aggregating stick table data from multiple HAProxy Enterprise nodes in real time and pushing that data back to each node, the profiling engine also stores historical data. For example, you can configure it to tell you what the average HTTP request rate was at the same time of day yesterday. Or, you can check what the average rate was at this same time a week ago, and adjust rate limiting to match.

Historical data allows you to perform dynamic decisions in your load balancer based on data from the past, such as to set rate limits that change depending on the hour.

Configure the Global Profiling Engine Jump to heading #

Follow these steps to configure historical aggregation of stick table data.

1. On the Global Profiling Engine server, as shown for real-time aggregation, configure the list of peers in the /etc/hapee-extras/hapee-gpe-stktagg.cfg file.

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations data
      from any to .aggregate
      peer profiling-engine 0.0.0.0:10000 local
      peer enterprise1 192.168.50.41:10000
      
      # list more 'peer' lines for other load balancers in the cluster
      # e.g. peer enterprise2 192.168.50.42:10000

   hapee-gpe-stktagg.cfg
   haproxy
   global
     stats socket /var/run/hapee-extras/gpe-api.sock
   aggregations data
      from any to .aggregate
      peer profiling-engine 0.0.0.0:10000 local
      peer enterprise1 192.168.50.41:10000
      
      # list more 'peer' lines for other load balancers in the cluster
      # e.g. peer enterprise2 192.168.50.42:10000

2. Edit the file /etc/hapee-extras/hapee-gpe.json to configure data retention policies for storing historical data.

   Data is stored in buckets of time. For example, you might keep 24 1-hour buckets, as shown below, which would allow you to compare a client’s current request rate to the average request rate during the same hour yesterday, for example.

   hapee-gpe.json
   json
   {
       "worker_thread_count": 4,
       "inter_worker_queue_size": 1024,
       "collector_queue_size": 64,
       "httpd_port": 9888,
       "datadir": "/var/cache/hapee-extras/hct_datadir",
       "default_stick_table_handling": 1,
       "ignore_tables": [],
       "detail_tables": [],
       "aggregate_tables": [],
       "stat_retentions": [
           {
               // 24 1-hour buckets
               "duration": 3600,
               "retention": 24
           }
       ]
   }

   hapee-gpe.json
   json
   {
       "worker_thread_count": 4,
       "inter_worker_queue_size": 1024,
       "collector_queue_size": 64,
       "httpd_port": 9888,
       "datadir": "/var/cache/hapee-extras/hct_datadir",
       "default_stick_table_handling": 1,
       "ignore_tables": [],
       "detail_tables": [],
       "aggregate_tables": [],
       "stat_retentions": [
           {
               // 24 1-hour buckets
               "duration": 3600,
               "retention": 24
           }
       ]
   }

   In this example:

   • The httpd_port field sets the port on which to publish historical data, which HAProxy Enterprise servers poll for updates. Here, it hosts the data at port 9888.

   • The aggregate_tables, detail_tables, and ignore_tables fields are all empty since we set default_stick_table_handling to 1 which will aggregate all tables.

   • The stat_retentions section lists data retention policies. Each policy sets a duration in seconds, which is the size of the data bucket, and a retention, which is the number of buckets to keep. A bucket stores counters from your stick tables. For example, if your stick table tracks the HTTP request rate over 10 seconds, a 1-hour bucket might store many thousands of these 10-second request rate data points.

   For each bucket, the server calculates statistics and serves them on the configured port.

3. Restart the Global Profiling Engine:

   nix
   sudo systemctl restart hapee-extras-gpe

   nix
   sudo systemctl restart hapee-extras-gpe

Configure HAProxy Enterprise Jump to heading #

Example: Use the Global Profiling Engine to enforce rate limits Jump to heading #

One use case for historical aggregation is to compare a client’s current request rates against the request rates over time and to then make rate-limiting decisions based on the current rate.

Configure each HAProxy Enterprise server to download and use the historical data.

1. Create an empty file at /etc/hapee-2.8/historical.map.

   Although an in-memory representation of this file will hold historical values received from the profiling engine, the file must exist on the filesystem when HAProxy Enterprise starts.

   HAProxy Enterprise updates a representation of this file in memory only. You will not see the contents of the file itself updated and it will remain empty, but you can see the in-memory values by calling the Runtime API show map method.

2. Install the Update module, which polls the profiling engine for new data to load into the map file.

   apt yum zypper pkg

   nix
   sudo apt-get install hapee-2.8r1-lb-update

   nix
   sudo apt-get install hapee-2.8r1-lb-update

   nix
   sudo yum install hapee-2.8r1-lb-update

   nix
   sudo yum install hapee-2.8r1-lb-update

   nix
   sudo zypper install hapee-2.8r1-lb-update

   nix
   sudo zypper install hapee-2.8r1-lb-update

   nix
   sudo pkg install hapee-2.8r1-lb-update

   nix
   sudo pkg install hapee-2.8r1-lb-update

3. Edit the file /etc/hapee-2.8/hapee-lb.cfg.

   In the global section of the file, add a module-load directive to load the Update module:

   hapee-lb.cfg
   haproxy
   global
     module-load hapee-lb-update.so

   hapee-lb.cfg
   haproxy
   global
     module-load hapee-lb-update.so

4. Configure the Update module to poll the profiling engine’s /aggs endpoint for data by adding a dynamic-update section that contains an update directive.

   The url parameter should use the profiling engine’s IP address.

   hapee-lb.cfg
   haproxy
   dynamic-update
     update id /etc/hapee-2.8/historical.map map url http://192.168.50.40:9888/aggs delay 3600s log

   hapee-lb.cfg
   haproxy
   dynamic-update
     update id /etc/hapee-2.8/historical.map map url http://192.168.50.40:9888/aggs delay 3600s log

   In this example:

   • The dynamic-update section configures HAProxy Enterprise to poll the profiling engine for historical data updates.
   • The update line’s id parameter sets the local file to update (remember, this file will not be updated on disk, only in HAProxy Enterprise’s runtime memory).
   • The map parameter switches the Update module into map file mode.
   • The url parameter specifies the IP and port of the profiling engine. It specifies the /aggs URL path.
   • The delay parameter sets the polling interval to 3600 seconds. Since our smallest stat_retentions duration is 3600 seconds, we can poll GPE hourly.
   • The log parameter enables logging to the HAProxy Enterprise access log.

5. As you would for real-time aggregation, add a peers section that lists the local node and the profiling engine on server lines.

   Here you will also define stick tables with their .aggregate clones.

   hapee-lb.cfg
   haproxy
   peers mypeers
     bind :10000
     server enterprise1
     server profiling-engine 192.168.50.40:10000
     table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
     table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

   hapee-lb.cfg
   haproxy
   peers mypeers
     bind :10000
     server enterprise1
     server profiling-engine 192.168.50.40:10000
     table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
     table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

   Be sure that the hostname of the HAProxy Enterprise node and the hostname of the Global Profiling Engine instance that you specify are the configured hostnames of those instances. Use the hostname command on each instance to retrieve the names.

   nix
   hostname

   nix
   hostname

   output
   text
   enterprise1

   output
   text
   enterprise1

6. Use map fetch methods in your frontend section to read information from the local map file and make traffic routing decisions.

   In the example below, we deny clients that have a request rate higher than the 99th percentile of requests from the same hour (3600 seconds) yesterday (86400 seconds ago).

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
     
     # add records to the stick table using the client's IP address as the table key
     http-request track-sc0 src table mypeers/request_rates
     
     # store the 99th percentile rate and the client's current rate in variables
     http-request set-var(req.rate_99percentile) str(/request_rates.http_req_rate.3600sec.86400sec_ago.99p),map(/etc/hapee-/historical.map,1000)
     http-request set-var(req.client_rate) sc_http_req_rate(0,mypeers/request_rates.aggregate)
     
     # set ACL expressions
     acl historical_rate_greater_than_zero var(req.rate_99percentile) -m int gt 0
     acl client_rate_exceeds_historical_rate var(req.rate_99percentile),sub(req.client_rate) -m int lt 0
     
     # deny the request if it exceeds the historical rate
     http-request deny deny_status 429 if historical_rate_greater_than_zero client_rate_exceeds_historical_rate
     
     default_backend webservers

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
     
     # add records to the stick table using the client's IP address as the table key
     http-request track-sc0 src table mypeers/request_rates
     
     # store the 99th percentile rate and the client's current rate in variables
     http-request set-var(req.rate_99percentile) str(/request_rates.http_req_rate.3600sec.86400sec_ago.99p),map(/etc/hapee-/historical.map,1000)
     http-request set-var(req.client_rate) sc_http_req_rate(0,mypeers/request_rates.aggregate)
     
     # set ACL expressions
     acl historical_rate_greater_than_zero var(req.rate_99percentile) -m int gt 0
     acl client_rate_exceeds_historical_rate var(req.rate_99percentile),sub(req.client_rate) -m int lt 0
     
     # deny the request if it exceeds the historical rate
     http-request deny deny_status 429 if historical_rate_greater_than_zero client_rate_exceeds_historical_rate
     
     default_backend webservers

   In this example:

   • The http-request track-sc0 line adds the current client to the stick table, using their source IP address as the primary key.

   • The http-request set-var(req.rate_99percentile) line reads the value of the /request_rates.http_req_rate.3600sec.86400sec_ago.99p statistic from the historical.map data. If that statistic does not exist or has no data (which happens if there was no traffic during that hour), a value of 1000 is used instead. See the Reference guide to learn how these statistics are named.

   • The http-request set-var(req.client_rate) line retrieves the current client’s request rate from the mypeers/request_rates.aggregate table, which uses real-time aggregation to collect data from all load balancers in your cluster.

   • The http-request deny line rejects requests if the client’s current request rate (aggregated across load balancers) exceeds the 99th percentile rate for all users from the same hour yesterday. (note: If the historical rate is zero, then it defaults to a value of 1000).

7. Restart HAProxy Enterprise.

   nix
   sudo systemctl restart hapee-2.8-lb

   nix
   sudo systemctl restart hapee-2.8-lb

Verify the setup. First, check that the HAProxy Enterprise admin logs show that the Update module is downloading the map file successfully. If there was an error, it will be written there. If everything worked, there will be no output (no errors).

Also, verify that data is being published by calling the /aggs URL with curl on the aggregation server. You will need to wait until the first bucket has been populated with data, though, which depends on the size of the bucket, before you will see data.

nix
curl http://localhost:9888/aggs

nix
curl http://localhost:9888/aggs

output
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 13
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 29
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 2
/request_rates.http_req_rate.3600sec.3600sec_ago.min 1
/request_rates.http_req_rate.3600sec.3600sec_ago.max 7
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 2
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 3

output
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 13
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 29
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 2
/request_rates.http_req_rate.3600sec.3600sec_ago.min 1
/request_rates.http_req_rate.3600sec.3600sec_ago.max 7
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 2
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 3

You can also call the Runtime API’s show map function to see the data stored in the map file.

Example: Use the Global Profiling Engine to calculate response time percentiles Jump to heading #

You can use the Global Profiling Engine to track response time percentiles across your HAProxy Enterprise cluster. You can record these response time percentiles on some interval and then use the data for analysis.

Configure each HAProxy Enterprise server to download and use the historical data.

1. Create an empty file at /etc/hapee-2.8/historical.map.

   Although an in-memory representation of this file will hold historical values received from the profiling engine, the file must exist on the filesystem when HAProxy Enterprise starts. HAProxy Enterprise updates a representation of this file in memory only. You will not see the contents of the file itself updated and it will remain empty, but you can see the in-memory values by calling the Runtime API show map method.

2. Install the Update module, which polls the profiling engine for new data to load into the map file.

   apt yum zypper pkg

   nix
   sudo apt-get install hapee-2.8r1-lb-update

   nix
   sudo apt-get install hapee-2.8r1-lb-update

   nix
   sudo yum install hapee-2.8r1-lb-update

   nix
   sudo yum install hapee-2.8r1-lb-update

   nix
   sudo zypper install hapee-2.8r1-lb-update

   nix
   sudo zypper install hapee-2.8r1-lb-update

   nix
   sudo pkg install hapee-2.8r1-lb-update

   nix
   sudo pkg install hapee-2.8r1-lb-update

3. Edit the file /etc/hapee-2.8/hapee-lb.cfg.

   In the global section of the file, add a module-load directive to load the Update module:

   hapee-lb.cfg
   haproxy
   global
     module-load hapee-lb-update.so

   hapee-lb.cfg
   haproxy
   global
     module-load hapee-lb-update.so

4. Configure the Update module to poll the profiling engine’s /aggs endpoint for data by adding a dynamic-update section that contains an update directive.

   The url parameter should use the profiling engine’s IP address.

   hapee-lb.cfg
   haproxy
   dynamic-update
     update id /etc/hapee-2.8/historical.map map url http://192.168.50.40:9888/aggs delay 10s log

   hapee-lb.cfg
   haproxy
   dynamic-update
     update id /etc/hapee-2.8/historical.map map url http://192.168.50.40:9888/aggs delay 10s log

   In this example:

   • The dynamic-update section configures HAProxy Enterprise to poll the profiling engine for historical data updates.
   • The update line’s id parameter sets the local file to update (remember, this file will not be updated on disk, only in HAProxy Enterprise’s runtime memory).
   • The map parameter switches the Update module into map file mode.
   • The url parameter specifies the IP and port of the profiling engine. It specifies the /aggs URL path.
   • The delay parameter sets the polling interval to 10 seconds.
   • The log parameter enables logging to the HAProxy Enterprise access log.

5. As you would for real-time aggregation, add a peers section that lists the local node and the profiling engine on server lines.

   Here you will also define stick tables with their .aggregate clones.

   hapee-lb.cfg
   haproxy
   peers mypeers
     bind :10000
     server enterprise1
     server profiling-engine 192.168.50.40:10000
     table st_responsetime type string len 64 size 100k expire 1h store gpt0
     table st_responsetime.aggregate type string len 64 size 100k expire 1h store gpt0

   hapee-lb.cfg
   haproxy
   peers mypeers
     bind :10000
     server enterprise1
     server profiling-engine 192.168.50.40:10000
     table st_responsetime type string len 64 size 100k expire 1h store gpt0
     table st_responsetime.aggregate type string len 64 size 100k expire 1h store gpt0

   Be sure that the hostname of the HAProxy Enterprise node and the hostname of the Global Profiling Engine instance that you specify are the configured hostnames of those instances. Use the hostname command on each instance to retrieve the names.

   nix
   hostname

   nix
   hostname

   output
   text
   enterprise1

   output
   text
   enterprise1

6. In your frontend, track the total response time of each request in the stick table mypeers/st_responsetime. We use the general purpose tag (gpt0) to store the response time value in the stick table. Each record uses a unique ID as its key, where the unique ID’s format is a combination of the client’s IP address, client’s port, frontend IP address, frontend port, a timestamp, a request counter, and the process ID.

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
      
     # generate a unique ID 
     unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
     http-request set-var(txn.path) path
     # add records to the stick table using the unique ID as table key
     http-request track-sc0 unique-id table mypeers/st_responsetime
     
     # prepare and perform the calculation for response times
     http-response set-var-fmt(txn.response_time) %Tr
     http-response set-var-fmt(txn.connect_time) %Tc
     http-response set-var-fmt(txn.queue_time) %Tw
     http-response sc-set-gpt0(0) var(txn.response_time),add(txn.queue_time),add(txn.connect_time)
     
     # store the 99th percentile rate in variables
     http-request set-var(req.response_time_99percentile) str(/st_responsetime.gpt0.3600sec.3600sec_ago.99p),map(/etc/hapee-2.8/historical.map,1000)
       
     default_backend webservers

   hapee-lb.cfg
   haproxy
   frontend fe_main
     bind :80
      
     # generate a unique ID 
     unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
     http-request set-var(txn.path) path
     # add records to the stick table using the unique ID as table key
     http-request track-sc0 unique-id table mypeers/st_responsetime
     
     # prepare and perform the calculation for response times
     http-response set-var-fmt(txn.response_time) %Tr
     http-response set-var-fmt(txn.connect_time) %Tc
     http-response set-var-fmt(txn.queue_time) %Tw
     http-response sc-set-gpt0(0) var(txn.response_time),add(txn.queue_time),add(txn.connect_time)
     
     # store the 99th percentile rate in variables
     http-request set-var(req.response_time_99percentile) str(/st_responsetime.gpt0.3600sec.3600sec_ago.99p),map(/etc/hapee-2.8/historical.map,1000)
       
     default_backend webservers

7. Restart HAProxy Enterprise.

   nix
   sudo systemctl restart hapee-2.8-lb

   nix
   sudo systemctl restart hapee-2.8-lb

Verify the setup. First, check that the HAProxy Enterprise admin logs show that the Update module is downloading the map file successfully. If there was an error, it will be written there. If everything worked, there will be no output (no errors).

Also, verify that data is being published by calling the /aggs URL with curl on the aggregation server. You will need to wait until the first bucket has been populated with data, though, which depends on the size of the bucket, before you will see data.

nix
curl http://localhost:9888/aggs

nix
curl http://localhost:9888/aggs

output
text
/st_responsetime.gpt0.3600sec.3600sec_ago.cnt 2
/st_responsetime.gpt0.3600sec.3600sec_ago.sum 24153
/st_responsetime.gpt0.3600sec.3600sec_ago.avg 12076
/st_responsetime.gpt0.3600sec.3600sec_ago.per_sec_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.burst_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.min 9769
/st_responsetime.gpt0.3600sec.3600sec_ago.max 14384
/st_responsetime.gpt0.3600sec.3600sec_ago.50p 9775
/st_responsetime.gpt0.3600sec.3600sec_ago.75p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.90p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.95p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99.9p 14391

output
text
/st_responsetime.gpt0.3600sec.3600sec_ago.cnt 2
/st_responsetime.gpt0.3600sec.3600sec_ago.sum 24153
/st_responsetime.gpt0.3600sec.3600sec_ago.avg 12076
/st_responsetime.gpt0.3600sec.3600sec_ago.per_sec_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.burst_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.min 9769
/st_responsetime.gpt0.3600sec.3600sec_ago.max 14384
/st_responsetime.gpt0.3600sec.3600sec_ago.50p 9775
/st_responsetime.gpt0.3600sec.3600sec_ago.75p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.90p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.95p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99.9p 14391

You can also call the Runtime API’s show map function to see the data stored in the map file.

Global Profiling Engine configuration file reference Jump to heading #

The following fields can be set in the /etc/hapee-extras/hapee-gpe.json file:

Global Profiling Engine statistics reference Jump to heading #

Calling the /aggs endpoint on port 9888 returns a list of available statistics. For example, if you set the following retention policy in the stat_retentions field:

json
// 24 1-hour buckets
"duration": 3600,
"retention": 24

json
// 24 1-hour buckets
"duration": 3600,
"retention": 24

The /aggs endpoint would return data for each bucket. Each bucket contains one hour of data (3600-seconds), representing one of the hours during the last 24 hours. For example, the following statistics are recorded for the hour that happened one hour ago:

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 362
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 3547
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.min 8
/request_rates.http_req_rate.3600sec.3600sec_ago.max 11
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 9

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 362
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 3547
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.min 8
/request_rates.http_req_rate.3600sec.3600sec_ago.max 11
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 9

Each line is a key and value. The key has this format:

/name-of-stick-table . name-of-counter . bucket-duration . time since bucket occurred . statistic

For example:

text
/request_rates.http_req_rate.3600sec.86400sec_ago.99p

text
/request_rates.http_req_rate.3600sec.86400sec_ago.99p

Some lines have a negative number in them:

text
/request_rates.http_req_rate.3600sec.-3600sec_ago.cnt

text
/request_rates.http_req_rate.3600sec.-3600sec_ago.cnt

This indicates a sliding time window that has a begin and end time that changes at a regular inteveral (i.e. an hour ago from now, or more realistically, at the next time the smallest bucket is calculated). In contrast, the following metric would be updated only at the top of every hour:

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt

The table below describes each statistic:

Troubleshooting: peer connections Jump to heading #

If calling the /aggs URL with curl on the aggregation server does not produce any data even after adequate time has passed for the first bucket to populate, it may be the case that the Global Profiling Engine does not have connection to its configured HAProxy Enterprise peers.

To verify peer connections:

1. Use the show peers Runtime API command on the HAProxy Enterprise server:

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.8/hapee-lb.sock

   output
   text
   0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
     0xaaaad88e2750: id=profiling-engine(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
           reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
           flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
           shared tables:
             0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
     0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
           reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
           flags=0x0
           shared tables:
             0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")

   output
   text
   0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
     0xaaaad88e2750: id=profiling-engine(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
           reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
           flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
           shared tables:
             0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
     0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
           reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
           flags=0x0
           shared tables:
             0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")

   There should be an entry for the Global Profiling engine, as well as all connected HAProxy Enterprise peers. If no data is returned or if peers are missing, there may be an issue with the configuration.

2. If entries are missing, check both the peers section of /etc/hapee-2.8/hapee-lb.cfg and the peers entries in the Global Profiling Engine configuration file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

   Be sure that the hostname of the HAProxy Enterprise node and the hostname of the Global Profiling Engine instance that you specify are the configured hostnames of those instances. Use the hostname command on each instance to retrieve the names.

   nix
   hostname

   nix
   hostname

   output
   text
   enterprise1

   output
   text
   enterprise1