Enterprise modules

Global Profiling Engine

On this page

The Global Profiling Engine collects stick table data from all HAProxy Enterprise nodes in a cluster. It can perform calculations on both current and historical data before pushing the data back to the load balancers, giving them all a comprehensive and up-to-date view of client behavior across the cluster.

For example, in an active-active load balancer configuration where traffic is relayed to two or more load balancers in a round-robin rotation, the GPE ensures that each load balancer receives the sum of requests that a client has made, even when some of those requests were routed to the other load balancer.

It calculates statistics for current, real-time traffic, which you can use to take action based on an immediate threat, such as a web scraper. It also calculates historical statistics, which you can use to make decisions based on traffic trends from the past. For example, you can factor in the request rate at the 90th percentile from yesterday.

The Global Profiling Engine should be installed on a separate server from your HAProxy Enterprise nodes, from where it can collect and aggregate data from stick tables from across your load balancer cluster.

When would you use the profiling engine? The table below outlines various, common ways to set up your load-balancing cluster and explains which benefit from using the Global Profiling Engine:

Standalone load balancer

If you operate a single HAProxy Enterprise node, collecting data with stick tables can help improve security and observability by monitoring client activity in real time. However, you do not need to use the Global Profiling Engine because there aren’t multiple load balancers collecting data that needs to be aggregated, unless you want to use its historical aggregations feature.

Active-standby cluster

In an active-standby cluster, you operate two HAProxy Enterprise nodes as peers, with one peer actively receiving traffic while the other is on standby. If you define a peers section in your configuration, then the stick tables of the active node are propagated to the passive node, where they overwrite existing values. Overwriting would be bad if both peers were receiving traffic and collecting data, but in an active-passive scenario, it’s perfectly fine. The metrics stay accurate on all nodes. In this case, you do not need the Global Profiling Engine, which aggregates data from multiple, active peers, unless you want to use its historical aggregations feature.

Active-active cluster with Global Profiling Engine

In an active-active cluster, you operate two HAProxy Enterprise nodes as peers, with both receiving traffic at the same time. To ensure that stick table counters are accurate cluster-wide, you need the Global Profiling Engine, which runs on a separate server, collecting stick table data from both peers and aggregating it. Once aggregated, the data is sent back to the peers so that they can make decisions based on it.

Installation Jump to heading

The following section gives detailed information on how to install the Global Profiling Engine.

  1. Provision a server on which you will install the Global Profiling Engine.

  2. On the new server, add package repositories for your operating system.

    In a terminal window, set a variable named key to your HAProxy Enterprise license key.

    nix
    key=<HAProxy Enterprise Key>
    nix
    key=<HAProxy Enterprise Key>

    Then use the echo and tee commands to add the HAProxy Technologies extras repository location into the file /etc/apt/sources.list.d/haproxy-tech.list, as shown below.

    nix
    echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/debian-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list
    nix
    echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/debian-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

    In a terminal window, set a variable named key to your HAProxy Enterprise license key.

    nix
    key=<HAProxy Enterprise Key>
    nix
    key=<HAProxy Enterprise Key>

    Then use the echo and tee commands to add the HAProxy Technologies extras repository location into the file /etc/apt/sources.list.d/haproxy-tech.list, as shown below.

    nix
    echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/ubuntu-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list
    nix
    echo "deb [arch=amd64] https://www.haproxy.com/download/hapee/key/${key}-plus/extras/ubuntu-$(. /etc/os-release && echo "$VERSION_CODENAME")/amd64/ $(. /etc/os-release && echo "$VERSION_CODENAME") main" | sudo tee -a /etc/apt/sources.list.d/haproxy-tech.list

    Create a new file /etc/yum.repos.d/haproxy-tech.repo if it does not exist and add the contents below. Replace <HAProxy Enterprise Key> with your HAProxy Enterprise license key. Also replace rhel-8 with your RHEL version number.

    haproxy-tech.repo
    ini
    [hapee-plus-extras]
    name=hapee-plus-extras
    enabled=1
    baseurl=https://www.haproxy.com/download/hapee/key/<HAProxy Enterprise Key>-plus/extras/rhel-8/$basearch/bin/
    gpgcheck=1
    haproxy-tech.repo
    ini
    [hapee-plus-extras]
    name=hapee-plus-extras
    enabled=1
    baseurl=https://www.haproxy.com/download/hapee/key/<HAProxy Enterprise Key>-plus/extras/rhel-8/$basearch/bin/
    gpgcheck=1

  3. Install packages:

    nix
    sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
    wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install --yes hapee-extras-gpe
    nix
    sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
    wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install --yes hapee-extras-gpe
    nix
    sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
    wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install --yes hapee-extras-gpe
    nix
    sudo apt-get install --yes apt-transport-https curl dirmngr gnupg-agent grep
    wget -O - https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install --yes hapee-extras-gpe
    nix
    rpm --import https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc
    yum makecache
    yum install -y hapee-extras-gpe
    nix
    rpm --import https://pks.haproxy.com/linux/enterprise/HAPEE-key-extras.asc
    yum makecache
    yum install -y hapee-extras-gpe

  4. To start the Global Profiling Engine, run:

    nix
    sudo systemctl enable hapee-extras-gpe
    sudo systemctl start hapee-extras-gpe
    nix
    sudo systemctl enable hapee-extras-gpe
    sudo systemctl start hapee-extras-gpe

Manage the service Jump to heading

Start, stop and check the status of the Global Profiling Engine service by using the corresponding systemctl commands.

Start the service:

nix
sudo systemctl start hapee-extras-gpe
nix
sudo systemctl start hapee-extras-gpe

Restart the service:

nix
sudo systemctl restart hapee-extras-gpe
nix
sudo systemctl restart hapee-extras-gpe

Show current status:

nix
sudo systemctl status hapee-extras-gpe
nix
sudo systemctl status hapee-extras-gpe

Stop the service:

nix
sudo systemctl stop hapee-extras-gpe
nix
sudo systemctl stop hapee-extras-gpe

Real-time aggregation Jump to heading

The Global Profiling Engine collects stick table data from all HAProxy Enterprise nodes in the cluster in real time. It then aggregates that data and pushes it back to all of the nodes. For example, if LoadBalancer1 receives two requests and LoadBalancer2 receives three requests, the Global Profiling Engine will sum those numbers to get a total of five, then push that to both LoadBalancer1 and LoadBalancer2. This is helpful for an active/active load balancer configuration wherein the nodes need to share client request information to form an accurate picture of activity across the cluster.

The aggregated data does not overwrite the data on the load balancer nodes. Instead, it is pushed to secondary stick tables that have, for example, a suffix of .aggregate. You would use a fetch method to retrieve the aggregated data and perform an action, such as rate limiting.

Stick table data is transferred between the HAProxy Enterprise servers and the Global Profiling Engine server by using the peers protocol, a protocol created specifically for this purpose. You must configure which servers should participate, both on the Global Profiling Engine server and on each HAProxy Enterprise node.

Configure HAProxy Enterprise nodes Jump to heading

An HAProxy Enterprise node must be configured to share its stick table data with the Global Profiling Engine server. Once aggregated, the profiling engine sends the data back to each node where it is stored in a new stick table. Follow these steps on each load balancer:

  1. Edit the file /etc/hapee-2.9/hapee-lb.cfg.

    Add a peers section.

    hapee-lb.cfg
    haproxy
    global
      [...]
      # By setting this, you are directing HAProxy Enterprise to use the server line
      # that specifies this name as the local node.
      localpeer enterprise1
      [...] 
     peers mypeers
       # This is the address and port that the load balancer will receive aggregated data from the GPE server
       bind 0.0.0.0:10000
       # The local HAProxy Enterprise node hostname defined by one of the following:
       # 1) the value provided when the load balancer process is started with the -L argument
       # 2) the localpeer name from the global section of the load balancer configuration (suggested method)
       # 3) the hostname as returned by the system hostname command (default)
       server enterprise1
      
       # The Global Profiling Engine
       # If you run GPE on the same server, use a different port here
       server gpe 192.168.50.40:10000
      
       # stick tables definitions
       table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
       table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)
    hapee-lb.cfg
    haproxy
    global
      [...]
      # By setting this, you are directing HAProxy Enterprise to use the server line
      # that specifies this name as the local node.
      localpeer enterprise1
      [...] 
     peers mypeers
       # This is the address and port that the load balancer will receive aggregated data from the GPE server
       bind 0.0.0.0:10000
       # The local HAProxy Enterprise node hostname defined by one of the following:
       # 1) the value provided when the load balancer process is started with the -L argument
       # 2) the localpeer name from the global section of the load balancer configuration (suggested method)
       # 3) the hostname as returned by the system hostname command (default)
       server enterprise1
      
       # The Global Profiling Engine
       # If you run GPE on the same server, use a different port here
       server gpe 192.168.50.40:10000
      
       # stick tables definitions
       table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
       table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

    Inside it:

    • Define a bind line to set the IP address and port at which this node should receive data back from the Global Profiling Engine server. In this example, the bind directive listens on all IP addresses at port 10000 and receives aggregated data.

    • Define a server line for the current load balancer server. The server name value is important because it must match the name you set in the Global Profiling Engine server’s configuration for the corresponding peer line. The hostname may be one of the following, in order of precedence:

      • the value provided with the -L argument specified on the command line used to start the load balancer process
      • the localpeer name specified in the global section of the load balancer configuration (this method is used in this example)
      • the host name returned by the system hostname command. This is the default, but we recommend using one of the other two methods

      In this example, the local HAProxy Enterprise node is listed with only its hostname, enterprise1. It is not necessary to specify its IP address and port.

    • Define a server line for the Global Profiling Engine server. Set its IP address and port. The name you set here is also important. It must match the corresponding peer line in the Global Profiling Engine server’s configuration.

    • Define stick tables. For each one, add a duplicate line where the table name has the suffix .aggregate. In this example, the non-aggregated stick table request_rates will store current HTTP request rates. The stick tables record the rate at which clients make requests over 10 seconds. We clear out stale records after 30 seconds by setting the expire parameter on the stick table. The type parameter sets the key for the table, which in this case is an IP address. The stick table request_rates.aggregate receives its data from the Global Profiling Engine. Its suffix, .aggregate, will match the the profiling engine’s configuration.

    localpeer definition

    If you receive an error for your load balancer configuration that looks like the following after specifying the name for your load balancer on the server line:

    text
    [WARNING]  (6125) : config : Removing incomplete section 'peers mypeers' (no peer named 'enterprise1')
    text
    [WARNING]  (6125) : config : Removing incomplete section 'peers mypeers' (no peer named 'enterprise1')

    Specify your hostname value for localpeer in your global section:

    hapee-lb.cfg
    haproxy
    global
      localpeer enterprise1
    hapee-lb.cfg
    haproxy
    global
      localpeer enterprise1

    This global setting is required in cases where your hostname (retrieved using the system hostname command) is different from your desired peer name. Be sure to update your GPE configuration to use the name you specify as the localpeer name, as well as update your load balancer configuration to reference that name on the server line for your load balancer in your peers section).

  2. Add directives to your frontend, backend or listen sections that populate the non-aggregated stick tables with data.

    Below, the http-request track-sc0 line adds request rate information for each client that connects to the load balancer, using the client’s source IP address (src) as the key in the stick table.

    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
      default_backend webservers
      
      # add records to the stick table using the client's
      # source IP address as the table key
      http-request track-sc0 src table mypeers/request_rates
    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
      default_backend webservers
      
      # add records to the stick table using the client's
      # source IP address as the table key
      http-request track-sc0 src table mypeers/request_rates

  3. Add directives that read the aggregated data returned from the Global Profiling Engine server. That data is stored in the table with the suffix .aggregate.

    Below, the http-request deny line rejects clients that have a request rate greater than 1000. The client’s request rate is an aggregate amount calculated from all active load balancers. Note that this line reads data from the request_rates.aggregate table.

    hapee-lb.cfg
    haproxy
    # perform actions like rate limiting
    http-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 }
    hapee-lb.cfg
    haproxy
    # perform actions like rate limiting
    http-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 }

  4. Restart HAProxy Enterprise.

    nix
    sudo systemctl restart hapee-2.9-lb
    nix
    sudo systemctl restart hapee-2.9-lb

Configure the Global Profiling Engine Jump to heading

The Global Profiling Engine server collects stick table data from HAProxy Enterprise load balancers in your cluster, but you must set which load balancers will be allowed to participate by listing them in the configuration file.

  1. On the Global Profiling Engine server, edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

    In the aggregations section, add a peer line for the Global Profiling Engine itself and for each HAProxy Enterprise node. Each peer’s name (e.g. enterprise1) should match the name you set in the HAProxy Enterprise configuration, since that is how the profiling engine validates the peer.

    Peer names

    Be sure that the peer names you specify in the GPE server’s configuration match exactly the names you specified in your load balancer configuration. For example, the following load balancer configuration sets the load balancer’s localpeer name to enterprise1 and we reference this name again in the peers section:

    hapee-lb.cfg
    haproxy
    global
      localpeer enterprise1
      [...]
    peers mypeers
      bind 0.0.0.0:10000
      server enterprise1
      [...]
    hapee-lb.cfg
    haproxy
    global
      localpeer enterprise1
      [...]
    peers mypeers
      bind 0.0.0.0:10000
      server enterprise1
      [...]

    As such, it must appear in the GPE server’s configuration as enterprise1 in order for GPE to make connection to the load balancer.

    hapee-gpe-stktagg.cfg
    haproxy
    global
      # Enables the Global Profiling Engine API
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations data
      # set how to map non-aggregated to aggregated stick tables
      from any to .aggregate
      
      # the profiling engine listens at this address
      peer gpe 0.0.0.0:10000 local
      
      # the load balancers listen at these addresses
      peer enterprise1 192.168.50.41:10000
      peer enterprise2 192.168.50.42:10000
    hapee-gpe-stktagg.cfg
    haproxy
    global
      # Enables the Global Profiling Engine API
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations data
      # set how to map non-aggregated to aggregated stick tables
      from any to .aggregate
      
      # the profiling engine listens at this address
      peer gpe 0.0.0.0:10000 local
      
      # the load balancers listen at these addresses
      peer enterprise1 192.168.50.41:10000
      peer enterprise2 192.168.50.42:10000

    In this example:

    • The Global Profiling Engine API provides a programmable API, which listens at the socket /var/run/hapee-extras/gpe-api.sock. The stats socket directive enables a CLI that lets you view data that the aggregator has stored.

    • In the aggregations section, the from line defines how non-aggregated stick tables map to aggregated stick tables, and what the suffix for the aggregated stick tables should be. The keyword any means that any stick table found will be aggregated. Aggregated data is pushed to tables with the same name, but ending with the suffix .aggregate. In the example, the engine expects stick tables to be named like request_rates and it will push aggregated data to request_rates.aggregate.

      You can also use a more specific mapping. In the example below, the engine expects stick tables to be named like request_rates.nonaggregate and it will push aggregated data to request_rates.aggregate. Stick tables without the .nonaggregate suffix will be ignored.

      haproxy
      from .nonaggregate to .aggregate
      haproxy
      from .nonaggregate to .aggregate

    • HAProxy Enterprise peer lines must use the same name you set on the server line in the HAProxy Enterprise configuration (e.g. enterprise1), and they must specify the IP addresses and ports where the load balancers are receiving aggregated data.

  2. Restart the Global Profiling Engine service:

    haproxy
    sudo systemctl restart hapee-extras-gpe
    haproxy
    sudo systemctl restart hapee-extras-gpe

Verify your setup Jump to heading

Check that the Global Profiling Engine and load balancers are setup correctly by utilizing their APIs.

  1. On the load balancer, call the Runtime API function show peers to check that the Global Profiling Engine is listed and that its last_status is ESTA (established):

    Below, the show peers command lists connected peers:

    nix
    echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock | head -2
    nix
    echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock | head -2
    output
    text
    0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=92
    0x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17s
    output
    text
    0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=92
    0x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17s

  2. Call the Runtime API function show table to see data in non-aggregated and aggregated stick tables.

    Below, we view data in the stick table named request_rates.aggregate:

    nix
    echo "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock
    nix
    echo "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock
    output
    text
    # table: mypeers/request_rates.aggregate, type: ip, size:102400, used:1
    0x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5
    output
    text
    # table: mypeers/request_rates.aggregate, type: ip, size:102400, used:1
    0x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5

  3. On the Global Profiling Engine server, call the show aggrs function to see load balancers that are registered as peers. A state of 0x7 means a successful connection. If you see a state of 0xffffffff, that means that a connection was not successful. Often, this is caused by the peer names not matching between the Global Profiling Engine’s configuration and the HAProxy Enterprise configuration.

    Below, the show aggrs command shows that the peer named enterprise1 has connected:

    nix
    echo "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.sock
    nix
    echo "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.sock
    output
    text
    aggregations data
      peer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3
             sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0
    output
    text
    aggregations data
      peer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3
             sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0

Optional: Bind outgoing connections to an interface Jump to heading

If the server where you are running the Global Profiling Engine has multiple network interfaces, you can configure the engine to bind to a specific one for outgoing data sent to HAProxy Enterprise servers.

To bind outgoing connections to a specific address, use the source directive in the global section.

IPv4 examples

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12:12345
hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12:12345

The port is optional. It defaults to 0 for random ports.

hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12
hapee-gpe-stktagg.cfg
haproxy
global
  source 126.123.10.12

IPv6 examples

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]:12345
hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]:12345

The port is optional. It defaults to 0 for random ports.

hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]
hapee-gpe-stktagg.cfg
haproxy
global
  source [2607:f8b0:400e:c00::ef]

GPE with session persistence Jump to heading

Available since

  • HAProxy Enterprise 2.9r1

A special situation arises when you want to use the Global Profiling Engine to sync session persistence data across load balancers. Session persistence uses a stick table to track which server a client was routed to initially and from then on continues to route that client to the same server.

  1. For example, consider the backend below that enables session persistence, but without GPE:

    haproxy
    backend servers
      stick-table type ip  size 1m  expire 30m
      stick on src
      server s1 192.168.0.10:80 check
      server s2 192.168.0.11:80 check
    haproxy
    backend servers
      stick-table type ip  size 1m  expire 30m
      stick on src
      server s1 192.168.0.10:80 check
      server s2 192.168.0.11:80 check

  2. We need to make the following changes to the backend:

    • Remove the stick-table line.
    • Make the stick on directive reference the sessions table in the peers section named mypeers.
    • By default, each load balancer can arrange the servers differently. However, we need to ensure consistent server IDs across all load balancers, so we use the id argument to set the IDs explicitly.
    haproxy
    backend servers
      stick on src table mypeers/sessions
      server s1 192.168.0.10:80 check id 1
      server s2 192.168.0.11:80 check id 2
    haproxy
    backend servers
      stick on src table mypeers/sessions
      server s1 192.168.0.10:80 check id 1
      server s2 192.168.0.11:80 check id 2

  3. Move the stick-table definition to the peers section:

    haproxy
    peers mypeers
      bind 0.0.0.0:10000
      server enterprise1
      server gpe 192.168.50.40:10000
      
      table sessions           type ip  size 1m  expire 30m store server_id,server_key
      table sessions.aggregate type ip  size 1m  expire 30m store server_id,server_key write-to mypeers/sessions
    haproxy
    peers mypeers
      bind 0.0.0.0:10000
      server enterprise1
      server gpe 192.168.50.40:10000
      
      table sessions           type ip  size 1m  expire 30m store server_id,server_key
      table sessions.aggregate type ip  size 1m  expire 30m store server_id,server_key write-to mypeers/sessions

    In this example:

    • We have moved the stick table to the peers section and named it sessions. You must set its store argument to server_id,server_key.
    • A table named sessions.aggregate syncs session persistence data to GPE, which then syncs it to all load balancers. The aggregate table must set the write-to argument so that the data is written back to the sessions table. The write-to parameter allows remote load balancers to update the local sessions table with session persistence data.

  4. Make this same change on the other load balancer.

Multi-level setup Jump to heading

You can aggregate stick tables from other Global Profiling Engines, which allows you to aggregate stick tables across different data centers, for example.

We will consider the following setup:

GPE multilevel diagram

The top-level aggr3 Global Profiling Engine will sum the counters from the intermediate aggr1 and aggr2 aggregate stick tables. It will then send the top-level aggregate stick table to all HAProxy Enterprise nodes.

You can also host multiple top-level servers for high availability. In that case, intermediate servers simply push their data to both. See below for details.

Configure the top-level Global Profiling Engine Jump to heading

Follow these steps on the server you wish to be the top-level Global Profiling Engine.

  1. Edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations toplevel
      from .intermediate to .aggregate
      peer top-profiling-engine  0.0.0.0:10000 local
      peer intermediate-profiling-engine1  192.168.56.111:10000 down
      peer intermediate-profiling-engine2  192.168.56.112:10000 down
    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations toplevel
      from .intermediate to .aggregate
      peer top-profiling-engine  0.0.0.0:10000 local
      peer intermediate-profiling-engine1  192.168.56.111:10000 down
      peer intermediate-profiling-engine2  192.168.56.112:10000 down

    • The current server has the local keyword set on its peer line.

    • In this example, two other Global Profiling Engine servers, intermediate-profiling-engine1 and intermediate-profiling-engine2, are listed with the down keyword, which means that they are one level down from the top.

    • The top-level Global Profiling Engine will aggregate stick table data from the intermediate servers. Their stick tables should have the .intermediate suffix.

    • The top-level Global Profiling Engine will push aggregated data back to the intermediate servers. The globally aggregated stick tables should have the .aggregate suffix.

Configure the intermediate Global Profiling Engines Jump to heading

Follow these steps on the servers you wish to be the intermediate-level Global Profiling Engines.

  1. Edit the file /etc/hapee-extras/hapee-gpe-stktagg.cfg.

    intermediate-profiling-engine1

    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations myaggr
      from any to .intermediate
      forward .aggregate
      peer intermediate-profiling-engine1  0.0.0.0:10000 local
      peer top-profiling-engine  192.168.56.113:10000 up
      peer enterprise1 192.168.50.41:10000
      peer enterprise2 192.168.50.42:10000
    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations myaggr
      from any to .intermediate
      forward .aggregate
      peer intermediate-profiling-engine1  0.0.0.0:10000 local
      peer top-profiling-engine  192.168.56.113:10000 up
      peer enterprise1 192.168.50.41:10000
      peer enterprise2 192.168.50.42:10000

    intermediate-profiling-engine2

    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations myaggr
      from any to .intermediate
      forward .aggregate
      peer intermediate-profiling-engine2  0.0.0.0:10000 local
      peer top-profiling-engine  192.168.56.113:10000 up
      peer enterprise3 192.168.50.51:10000
      peer enterprise4 192.168.50.52:10000
    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations myaggr
      from any to .intermediate
      forward .aggregate
      peer intermediate-profiling-engine2  0.0.0.0:10000 local
      peer top-profiling-engine  192.168.56.113:10000 up
      peer enterprise3 192.168.50.51:10000
      peer enterprise4 192.168.50.52:10000

    • The current server has the local keyword set on its peer line.

    • The upper-level Global Profiling Engine peer is denoted by the up keyword.

    • Each intermediate Global Profiling Engine is aware of only the HAProxy Enterprise nodes it manages and of the top-level Global Profiling Engine.

    • The intermediate-level Global Profiling Engines will aggregate stick table data from the HAProxy Enterprise servers.

    • The forward line relays the top-level server’s .aggregate stick tables to the HAProxy Enterprise servers.

    • The intermediate-level Global Profiling Engines will push aggregated data back to the HAProxy Enterprise servers. The aggregated stick tables should have the .aggregate suffix.

Configure for high availability Jump to heading

To create a highly available setup, you can have multiple top-level servers. Add the group parameter to them so that the intermediate servers recognize that there are two top-level servers.

intermediate-profiling-engine1

hapee-gpe-stktagg.cfg
haproxy
global
  stats socket /var/run/hapee-extras/gpe-api.sock
aggregations myaggr
  from any to .intermediate
  forward .aggregate
  peer intermediate-profiling-engine1  0.0.0.0:10000 local
  peer top-profiling-engine1  192.168.56.113:10000 up group 1
  peer top-profiling-engine2  192.168.56.114:10000 up group 1
  peer enterprise1 192.168.50.41:10000
  peer enterprise2 192.168.50.42:10000
hapee-gpe-stktagg.cfg
haproxy
global
  stats socket /var/run/hapee-extras/gpe-api.sock
aggregations myaggr
  from any to .intermediate
  forward .aggregate
  peer intermediate-profiling-engine1  0.0.0.0:10000 local
  peer top-profiling-engine1  192.168.56.113:10000 up group 1
  peer top-profiling-engine2  192.168.56.114:10000 up group 1
  peer enterprise1 192.168.50.41:10000
  peer enterprise2 192.168.50.42:10000

Historical aggregation Jump to heading

In addition to aggregating stick table data from multiple HAProxy Enterprise nodes in real time and pushing that data back to each node, the profiling engine also stores historical data. For example, you can configure it to tell you what the average HTTP request rate was at the same time of day yesterday. Or, you can check what the average rate was at this same time a week ago, and adjust rate limiting to match.

Historical data allows you to perform dynamic decisions in your load balancer based on data from the past, such as to set rate limits that change depending on the hour.

Configure the Global Profiling Engine Jump to heading

Follow these steps to configure historical aggregation of stick table data.

  1. On the Global Profiling Engine server, as shown for real-time aggregation, configure the list of peers in the /etc/hapee-extras/hapee-gpe-stktagg.cfg file.

    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations data
       from any to .aggregate
       peer profiling-engine 0.0.0.0:10000 local
       peer enterprise1 192.168.50.41:10000
       
       # list more 'peer' lines for other load balancers in the cluster
       # e.g. peer enterprise2 192.168.50.42:10000
    hapee-gpe-stktagg.cfg
    haproxy
    global
      stats socket /var/run/hapee-extras/gpe-api.sock
    aggregations data
       from any to .aggregate
       peer profiling-engine 0.0.0.0:10000 local
       peer enterprise1 192.168.50.41:10000
       
       # list more 'peer' lines for other load balancers in the cluster
       # e.g. peer enterprise2 192.168.50.42:10000

  2. Edit the file /etc/hapee-extras/hapee-gpe.json to configure data retention policies for storing historical data.

    Data is stored in buckets of time. For example, you might keep 24 1-hour buckets, as shown below, which would allow you to compare a client’s current request rate to the average request rate during the same hour yesterday, for example.

    hapee-gpe.json
    json
    {
        "worker_thread_count": 4,
        "inter_worker_queue_size": 1024,
        "collector_queue_size": 64,
        "httpd_port": 9888,
        "datadir": "/var/cache/hapee-extras/hct_datadir",
        "default_stick_table_handling": 1,
        "ignore_tables": [],
        "detail_tables": [],
        "aggregate_tables": [],
        "stat_retentions": [
            {
                // 24 1-hour buckets
                "duration": 3600,
                "retention": 24
            }
        ]
    }
    hapee-gpe.json
    json
    {
        "worker_thread_count": 4,
        "inter_worker_queue_size": 1024,
        "collector_queue_size": 64,
        "httpd_port": 9888,
        "datadir": "/var/cache/hapee-extras/hct_datadir",
        "default_stick_table_handling": 1,
        "ignore_tables": [],
        "detail_tables": [],
        "aggregate_tables": [],
        "stat_retentions": [
            {
                // 24 1-hour buckets
                "duration": 3600,
                "retention": 24
            }
        ]
    }

    In this example:

    • The httpd_port field sets the port on which to publish historical data, which HAProxy Enterprise servers poll for updates. Here, it hosts the data at port 9888.

    • The aggregate_tables, detail_tables, and ignore_tables fields are all empty since we set default_stick_table_handling to 1 which will aggregate all tables.

    • The stat_retentions section lists data retention policies. Each policy sets a duration in seconds, which is the size of the data bucket, and a retention, which is the number of buckets to keep. A bucket stores counters from your stick tables. For example, if your stick table tracks the HTTP request rate over 10 seconds, a 1-hour bucket might store many thousands of these 10-second request rate data points.

    For each bucket, the server calculates statistics and serves them on the configured port.

  3. Restart the Global Profiling Engine:

    nix
    sudo systemctl restart hapee-extras-gpe
    nix
    sudo systemctl restart hapee-extras-gpe

Configure HAProxy Enterprise Jump to heading

Example: Use the Global Profiling Engine to enforce rate limits Jump to heading

One use case for historical aggregation is to compare a client’s current request rates against the request rates over time and to then make rate-limiting decisions based on the current rate.

Configure each HAProxy Enterprise server to download and use the historical data.

  1. Create an empty file at /etc/hapee-2.9/historical.map.

    Although an in-memory representation of this file will hold historical values received from the profiling engine, the file must exist on the filesystem when HAProxy Enterprise starts.

    HAProxy Enterprise updates a representation of this file in memory only. You will not see the contents of the file itself updated and it will remain empty, but you can see the in-memory values by calling the Runtime API show map method.

  2. Install the Update module, which polls the profiling engine for new data to load into the map file.

    nix
    sudo apt-get install hapee-2.9r1-lb-update
    nix
    sudo apt-get install hapee-2.9r1-lb-update
    nix
    sudo yum install hapee-2.9r1-lb-update
    nix
    sudo yum install hapee-2.9r1-lb-update
    nix
    sudo zypper install hapee-2.9r1-lb-update
    nix
    sudo zypper install hapee-2.9r1-lb-update
    nix
    sudo pkg install hapee-2.9r1-lb-update
    nix
    sudo pkg install hapee-2.9r1-lb-update

  3. Edit the file /etc/hapee-2.9/hapee-lb.cfg.

    In the global section of the file, add a module-load directive to load the Update module:

    hapee-lb.cfg
    haproxy
    global
      module-load hapee-lb-update.so
    hapee-lb.cfg
    haproxy
    global
      module-load hapee-lb-update.so

  4. Configure the Update module to poll the profiling engine’s /aggs endpoint for data by adding a dynamic-update section that contains an update directive.

    The url parameter should use the profiling engine’s IP address.

    hapee-lb.cfg
    haproxy
    dynamic-update
      update id /etc/hapee-2.9/historical.map map url http://192.168.50.40:9888/aggs delay 3600s log
    hapee-lb.cfg
    haproxy
    dynamic-update
      update id /etc/hapee-2.9/historical.map map url http://192.168.50.40:9888/aggs delay 3600s log

    In this example:

    • The dynamic-update section configures HAProxy Enterprise to poll the profiling engine for historical data updates.
    • The update line’s id parameter sets the local file to update (remember, this file will not be updated on disk, only in HAProxy Enterprise’s runtime memory).
    • The map parameter switches the Update module into map file mode.
    • The url parameter specifies the IP and port of the profiling engine. It specifies the /aggs URL path.
    • The delay parameter sets the polling interval to 3600 seconds. Since our smallest stat_retentions duration is 3600 seconds, we can poll GPE hourly.
    • The log parameter enables logging to the HAProxy Enterprise access log.

  5. As you would for real-time aggregation, add a peers section that lists the local node and the profiling engine on server lines.

    Here you will also define stick tables with their .aggregate clones.

    hapee-lb.cfg
    haproxy
    peers mypeers
      bind :10000
      server enterprise1
      server profiling-engine 192.168.50.40:10000
      table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
      table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)
    hapee-lb.cfg
    haproxy
    peers mypeers
      bind :10000
      server enterprise1
      server profiling-engine 192.168.50.40:10000
      table request_rates           type ip size 100k expire 30s store http_req_rate(10s)
      table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)

    Be sure that the hostname of the HAProxy Enterprise node and the hostname of the Global Profiling Engine instance that you specify are the configured hostnames of those instances. Use the hostname command on each instance to retrieve the names.

    nix
    hostname
    nix
    hostname
    output
    text
    enterprise1
    output
    text
    enterprise1

  6. Use map fetch methods in your frontend section to read information from the local map file and make traffic routing decisions.

    In the example below, we deny clients that have a request rate higher than the 99th percentile of requests from the same hour (3600 seconds) yesterday (86400 seconds ago).

    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
      
      # add records to the stick table using the client's IP address as the table key
      http-request track-sc0 src table mypeers/request_rates
      
      # store the 99th percentile rate and the client's current rate in variables
      http-request set-var(req.rate_99percentile) str(/request_rates.http_req_rate.3600sec.86400sec_ago.99p),map(/etc/hapee-2.9/historical.map,1000)
      http-request set-var(req.client_rate) sc_http_req_rate(0,mypeers/request_rates.aggregate)
      
      # set ACL expressions
      acl historical_rate_greater_than_zero var(req.rate_99percentile) -m int gt 0
      acl client_rate_exceeds_historical_rate var(req.rate_99percentile),sub(req.client_rate) -m int lt 0
      
      # deny the request if it exceeds the historical rate
      http-request deny deny_status 429 if historical_rate_greater_than_zero client_rate_exceeds_historical_rate
      
      default_backend webservers
    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
      
      # add records to the stick table using the client's IP address as the table key
      http-request track-sc0 src table mypeers/request_rates
      
      # store the 99th percentile rate and the client's current rate in variables
      http-request set-var(req.rate_99percentile) str(/request_rates.http_req_rate.3600sec.86400sec_ago.99p),map(/etc/hapee-2.9/historical.map,1000)
      http-request set-var(req.client_rate) sc_http_req_rate(0,mypeers/request_rates.aggregate)
      
      # set ACL expressions
      acl historical_rate_greater_than_zero var(req.rate_99percentile) -m int gt 0
      acl client_rate_exceeds_historical_rate var(req.rate_99percentile),sub(req.client_rate) -m int lt 0
      
      # deny the request if it exceeds the historical rate
      http-request deny deny_status 429 if historical_rate_greater_than_zero client_rate_exceeds_historical_rate
      
      default_backend webservers

    In this example:

    • The http-request track-sc0 line adds the current client to the stick table, using their source IP address as the primary key.

    • The http-request set-var(req.rate_99percentile) line reads the value of the /request_rates.http_req_rate.3600sec.86400sec_ago.99p statistic from the historical.map data. If that statistic does not exist or has no data (which happens if there was no traffic during that hour), a value of 1000 is used instead. See the Reference guide to learn how these statistics are named.

    • The http-request set-var(req.client_rate) line retrieves the current client’s request rate from the mypeers/request_rates.aggregate table, which uses real-time aggregation to collect data from all load balancers in your cluster.

    • The http-request deny line rejects requests if the client’s current request rate (aggregated across load balancers) exceeds the 99th percentile rate for all users from the same hour yesterday. (note: If the historical rate is zero, then it defaults to a value of 1000).

  7. Restart HAProxy Enterprise.

    nix
    sudo systemctl restart hapee-2.9-lb
    nix
    sudo systemctl restart hapee-2.9-lb

Verify the setup. First, check that the HAProxy Enterprise admin logs show that the Update module is downloading the map file successfully. If there was an error, it will be written there. If everything worked, there will be no output (no errors).

Also, verify that data is being published by calling the /aggs URL with curl on the aggregation server. You will need to wait until the first bucket has been populated with data, though, which depends on the size of the bucket, before you will see data.

nix
curl http://localhost:9888/aggs
nix
curl http://localhost:9888/aggs
output
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 13
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 29
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 2
/request_rates.http_req_rate.3600sec.3600sec_ago.min 1
/request_rates.http_req_rate.3600sec.3600sec_ago.max 7
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 2
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 3
output
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 13
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 29
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 2
/request_rates.http_req_rate.3600sec.3600sec_ago.min 1
/request_rates.http_req_rate.3600sec.3600sec_ago.max 7
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 2
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 3
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 3

You can also call the Runtime API’s show map function to see the data stored in the map file.

Example: Use the Global Profiling Engine to calculate response time percentiles Jump to heading

You can use the Global Profiling Engine to track response time percentiles across your HAProxy Enterprise cluster. You can record these response time percentiles on some interval and then use the data for analysis.

Configure each HAProxy Enterprise server to download and use the historical data.

  1. Create an empty file at /etc/hapee-2.9/historical.map.

    Although an in-memory representation of this file will hold historical values received from the profiling engine, the file must exist on the filesystem when HAProxy Enterprise starts. HAProxy Enterprise updates a representation of this file in memory only. You will not see the contents of the file itself updated and it will remain empty, but you can see the in-memory values by calling the Runtime API show map method.

  2. Install the Update module, which polls the profiling engine for new data to load into the map file.

    nix
    sudo apt-get install hapee-2.9r1-lb-update
    nix
    sudo apt-get install hapee-2.9r1-lb-update
    nix
    sudo yum install hapee-2.9r1-lb-update
    nix
    sudo yum install hapee-2.9r1-lb-update
    nix
    sudo zypper install hapee-2.9r1-lb-update
    nix
    sudo zypper install hapee-2.9r1-lb-update
    nix
    sudo pkg install hapee-2.9r1-lb-update
    nix
    sudo pkg install hapee-2.9r1-lb-update

  3. Edit the file /etc/hapee-2.9/hapee-lb.cfg.

    In the global section of the file, add a module-load directive to load the Update module:

    hapee-lb.cfg
    haproxy
    global
      module-load hapee-lb-update.so
    hapee-lb.cfg
    haproxy
    global
      module-load hapee-lb-update.so

  4. Configure the Update module to poll the profiling engine’s /aggs endpoint for data by adding a dynamic-update section that contains an update directive.

    The url parameter should use the profiling engine’s IP address.

    hapee-lb.cfg
    haproxy
    dynamic-update
      update id /etc/hapee-2.9/historical.map map url http://192.168.50.40:9888/aggs delay 10s log
    hapee-lb.cfg
    haproxy
    dynamic-update
      update id /etc/hapee-2.9/historical.map map url http://192.168.50.40:9888/aggs delay 10s log

    In this example:

    • The dynamic-update section configures HAProxy Enterprise to poll the profiling engine for historical data updates.
    • The update line’s id parameter sets the local file to update (remember, this file will not be updated on disk, only in HAProxy Enterprise’s runtime memory).
    • The map parameter switches the Update module into map file mode.
    • The url parameter specifies the IP and port of the profiling engine. It specifies the /aggs URL path.
    • The delay parameter sets the polling interval to 10 seconds.
    • The log parameter enables logging to the HAProxy Enterprise access log.

  5. As you would for real-time aggregation, add a peers section that lists the local node and the profiling engine on server lines.

    Here you will also define stick tables with their .aggregate clones.

    hapee-lb.cfg
    haproxy
    peers mypeers
      bind :10000
      server enterprise1
      server profiling-engine 192.168.50.40:10000
      table st_responsetime type string len 64 size 100k expire 1h store gpt0
      table st_responsetime.aggregate type string len 64 size 100k expire 1h store gpt0
    hapee-lb.cfg
    haproxy
    peers mypeers
      bind :10000
      server enterprise1
      server profiling-engine 192.168.50.40:10000
      table st_responsetime type string len 64 size 100k expire 1h store gpt0
      table st_responsetime.aggregate type string len 64 size 100k expire 1h store gpt0

    Be sure that the hostname of the HAProxy Enterprise node and the hostname of the Global Profiling Engine instance that you specify are the configured hostnames of those instances. Use the hostname command on each instance to retrieve the names.

    nix
    hostname
    nix
    hostname
    output
    text
    enterprise1
    output
    text
    enterprise1

  6. In your frontend, track the total response time of each request in the stick table mypeers/st_responsetime. We use the general purpose tag (gpt0) to store the response time value in the stick table. Each record uses a unique ID as its key, where the unique ID’s format is a combination of the client’s IP address, client’s port, frontend IP address, frontend port, a timestamp, a request counter, and the process ID.

    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
       
      # generate a unique ID 
      unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
      http-request set-var(txn.path) path
      # add records to the stick table using the unique ID as table key
      http-request track-sc0 unique-id table mypeers/st_responsetime
      
      # prepare and perform the calculation for response times
      http-response set-var-fmt(txn.response_time) %Tr
      http-response set-var-fmt(txn.connect_time) %Tc
      http-response set-var-fmt(txn.queue_time) %Tw
      http-response sc-set-gpt0(0) var(txn.response_time),add(txn.queue_time),add(txn.connect_time)
      
      # store the 99th percentile rate in variables
      http-request set-var(req.response_time_99percentile) str(/st_responsetime.gpt0.3600sec.3600sec_ago.99p),map(/etc/hapee-2.9/historical.map,1000)
        
      default_backend webservers
    hapee-lb.cfg
    haproxy
    frontend fe_main
      bind :80
       
      # generate a unique ID 
      unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
      http-request set-var(txn.path) path
      # add records to the stick table using the unique ID as table key
      http-request track-sc0 unique-id table mypeers/st_responsetime
      
      # prepare and perform the calculation for response times
      http-response set-var-fmt(txn.response_time) %Tr
      http-response set-var-fmt(txn.connect_time) %Tc
      http-response set-var-fmt(txn.queue_time) %Tw
      http-response sc-set-gpt0(0) var(txn.response_time),add(txn.queue_time),add(txn.connect_time)
      
      # store the 99th percentile rate in variables
      http-request set-var(req.response_time_99percentile) str(/st_responsetime.gpt0.3600sec.3600sec_ago.99p),map(/etc/hapee-2.9/historical.map,1000)
        
      default_backend webservers

  7. Restart HAProxy Enterprise.

    nix
    sudo systemctl restart hapee-2.9-lb
    nix
    sudo systemctl restart hapee-2.9-lb

Verify the setup. First, check that the HAProxy Enterprise admin logs show that the Update module is downloading the map file successfully. If there was an error, it will be written there. If everything worked, there will be no output (no errors).

Also, verify that data is being published by calling the /aggs URL with curl on the aggregation server. You will need to wait until the first bucket has been populated with data, though, which depends on the size of the bucket, before you will see data.

nix
curl http://localhost:9888/aggs
nix
curl http://localhost:9888/aggs
output
text
/st_responsetime.gpt0.3600sec.3600sec_ago.cnt 2
/st_responsetime.gpt0.3600sec.3600sec_ago.sum 24153
/st_responsetime.gpt0.3600sec.3600sec_ago.avg 12076
/st_responsetime.gpt0.3600sec.3600sec_ago.per_sec_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.burst_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.min 9769
/st_responsetime.gpt0.3600sec.3600sec_ago.max 14384
/st_responsetime.gpt0.3600sec.3600sec_ago.50p 9775
/st_responsetime.gpt0.3600sec.3600sec_ago.75p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.90p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.95p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99.9p 14391
output
text
/st_responsetime.gpt0.3600sec.3600sec_ago.cnt 2
/st_responsetime.gpt0.3600sec.3600sec_ago.sum 24153
/st_responsetime.gpt0.3600sec.3600sec_ago.avg 12076
/st_responsetime.gpt0.3600sec.3600sec_ago.per_sec_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.burst_avg 0
/st_responsetime.gpt0.3600sec.3600sec_ago.min 9769
/st_responsetime.gpt0.3600sec.3600sec_ago.max 14384
/st_responsetime.gpt0.3600sec.3600sec_ago.50p 9775
/st_responsetime.gpt0.3600sec.3600sec_ago.75p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.90p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.95p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99p 14391
/st_responsetime.gpt0.3600sec.3600sec_ago.99.9p 14391

You can also call the Runtime API’s show map function to see the data stored in the map file.

Global Profiling Engine configuration file reference Jump to heading

The following fields can be set in the /etc/hapee-extras/hapee-gpe.json file:

Field Description
worker_thread_count The number of worker threads to start. Default: 2.
inter_worker_queue_size The size of the message queue that handles communication between workers. It must be greater than 2 and a power of 2. Default: 1024.
collector_queue_size The size of the message queue between workers and the collector. It must be greater than 2 and a power of 2. Default: 64.
httpd_port The TCP port on which to publish historical statistics data. Default: 9888.
datadir The directory in which to store historical data files.
default_stick_table_handling Indicates how the server should process stick tables that are not listed in the ignore_tables, detail_tables, or aggregate_tables arrays. Values: 0 = ignore, 1 = aggregate, 2 = detailed processing (Experimental).
aggregate_tables An array of stick table names that should be processed. One set of aggregates will be created for each stick table as a whole.
ignore_tables An array of stick table names that should be skipped during processing.
detail_tables (Experimental) An array of stick table names that should be processed. One set of aggregates will be created for every value in the stick table.
stat_retentions An array of data retention policies. Each policy should have: duration: an integer value in seconds indicating the size of the bucket (i.e. time period) to aggregate data for. retention: the number of buckets to keep.

Global Profiling Engine statistics reference Jump to heading

Calling the /aggs endpoint on port 9888 returns a list of available statistics. For example, if you set the following retention policy in the stat_retentions field:

json
// 24 1-hour buckets
"duration": 3600,
"retention": 24
json
// 24 1-hour buckets
"duration": 3600,
"retention": 24

The /aggs endpoint would return data for each bucket. Each bucket contains one hour of data (3600-seconds), representing one of the hours during the last 24 hours. For example, the following statistics are recorded for the hour that happened one hour ago:

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 362
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 3547
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.min 8
/request_rates.http_req_rate.3600sec.3600sec_ago.max 11
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 9
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt 362
/request_rates.http_req_rate.3600sec.3600sec_ago.sum 3547
/request_rates.http_req_rate.3600sec.3600sec_ago.avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.per_sec_avg 0
/request_rates.http_req_rate.3600sec.3600sec_ago.burst_avg 10
/request_rates.http_req_rate.3600sec.3600sec_ago.min 8
/request_rates.http_req_rate.3600sec.3600sec_ago.max 11
/request_rates.http_req_rate.3600sec.3600sec_ago.50p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.75p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.90p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.95p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99p 9
/request_rates.http_req_rate.3600sec.3600sec_ago.99.9p 9

Each line is a key and value. The key has this format:

/name-of-stick-table . name-of-counter . bucket-duration . time since bucket occurred . statistic

For example:

text
/request_rates.http_req_rate.3600sec.86400sec_ago.99p
text
/request_rates.http_req_rate.3600sec.86400sec_ago.99p

Some lines have a negative number in them:

text
/request_rates.http_req_rate.3600sec.-3600sec_ago.cnt
text
/request_rates.http_req_rate.3600sec.-3600sec_ago.cnt

This indicates a sliding time window that has a begin and end time that changes at a regular inteveral (i.e. an hour ago from now, or more realistically, at the next time the smallest bucket is calculated). In contrast, the following metric would be updated only at the top of every hour:

text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt
text
/request_rates.http_req_rate.3600sec.3600sec_ago.cnt

The table below describes each statistic:

Field Description
cnt The count of data points of the counter (e.g. HTTP rate limit) recorded in the bucket.
sum The sum of all data point values in the bucket.
avg An average of all data points in the bucket that preserves the time period of the stick table counter, which makes it easy to work with when comparing it to current request rates in HAProxy Enterprise; the sum of all data points (e.g. 3547) is multiplied by the stick table counter period (e.g. 10 for http_req_rate(10s)), then divided by the duration of the bucket (e.g. 3600).
persec_avg An average of all data points in the bucket, converted to a 1-second average (discards the period of the stick table counter).
burst_avg A traditional, mathematical average; the sum of all data points is divided by the count.
min The minimum data point value in the bucket.
max The maximum data point value in the bucket.
50p The 50th percentile.
75p The 75th percentile.
90p The 90th percentile.
95p The 95th percentile.
99p The 99th percentile.
99.9p The 99.9th percentile.

Troubleshooting: peer connections Jump to heading

If calling the /aggs URL with curl on the aggregation server does not produce any data even after adequate time has passed for the first bucket to populate, it may be the case that the Global Profiling Engine does not have connection to its configured HAProxy Enterprise peers.

To verify peer connections:

  1. Use the show peers Runtime API command on the HAProxy Enterprise server:

    nix
    echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock
    nix
    echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-2.9/hapee-lb.sock
    output
    text
    0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
      0xaaaad88e2750: id=profiling-engine(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
            reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
            flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
            shared tables:
              0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                  last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                  table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
              0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
      0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
            reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
            flags=0x0
            shared tables:
              0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
              0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
    output
    text
    0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
      0xaaaad88e2750: id=profiling-engine(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
            reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
            flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
            shared tables:
              0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                  last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                  table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
              0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
      0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
            reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
            flags=0x0
            shared tables:
              0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
            Dictionary cache not dumped (use "show peers dict")
              0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                  last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                  table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
            Dictionary cache not dumped (use "show peers dict")

    There should be an entry for the Global Profiling engine, as well as all connected HAProxy Enterprise peers. If no data is returned or if peers are missing, there may be an issue with the configuration.

  2. If entries are missing, check both the peers section of /etc/hapee-2.9/hapee-lb.cfg and the peers entries in the Global Profiling Engine configuration file /etc/hapee-extras/hapee-gpe-stktagg.cfg. Check the following:

    • The hostname for the Global Profiling Engine server should be its configured hostname. Use the hostname command to retrieve its name.

      nix
      hostname
      nix
      hostname
      output
      text
      gpe
      output
      text
      gpe

    • In each of the configuration files, /etc/hapee-2.9/hapee-lb.cfg, for the load balancer configuration, and /etc/hapee-extras/hapee-gpe-stktagg.cfg, for the Global Profiling Engine configuration, verify that the hostnames for your load balancer servers are identical everywhere they are referenced.

      The hostname value you provide for the load balancers may be one of three things:

      • the value provided with the -L argument specified on the command line used to start the load balancer process
      • the localpeer name specified in the global section of the load balancer configuration
      • the host name returned by the system hostname command.

      Be sure that this value is the one you specify for as the hostname of the load balancer on the server line in the peers section of your load balancer configuration and on the corresponding peer entry in the aggregations section of the GPE configuration.

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?

Previous page MaxMind Next page GSLB
© 2024 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences