About the Stick Table Aggregator

The HAProxy Enterprise Stick Table Aggregator leverages HAProxy Enterprise stick tables to provide real-time, cluster-wide tracking of network activity.

Stick tables

HAProxy Enterprise instances store client session stickiness information as key-value structures in stick tables. Typical stick table keys are client IP addresses, and client identifiers (32 bits integers).

Soft and hard restarts

Current stick table entries are reloaded when an HAProxy Enterprise peer is restarted.

Soft restart

The HAProxy Enterprise peer that restarts replicates relevant stick table content from the local shutting-down HAProxy Enterprise process.

Hard restart

The HAProxy Enterprise peer that restarts replicates relevant stick table content from remote peers, if any.

See also

The HAProxy Configuration Manual

The Stick Table Aggregator

The Stick Table Aggregator stores stick table entries, just like HAProxy Enterprise instances. It sends aggregate entries in a distinct stick table that it sends back to the HAProxy Enterprise peers it listens to.

HAProxy Enterprise peers then store a local, unsynchronized stick table, and an aggregate stick table provided by the Stick Table Aggregator.

The Stick Table Aggregator supports peers protocol versions 2.0 and 2.1.

Installing the Stick Table Aggregator

Install the Stick Table Aggregator using your packaging manager.

To install the Stick Table Aggregator on Debian based systems, run:

$ sudo apt-get install hapee-extras-stktagg20

Managing the Stick Table Aggregator process

To start, stop, restart the Stick Table Aggregator, or display its status:

$ sudo systemctl <start|stop|restart|status> hapee-extras-stktagg

Running the Stick Table Aggregator

You can run the /opt/hapee-extras/sbin/hapee-stktagg command with the following options:

Option

Description

Notes

-h

Displays help.

-d

Enters debug mode.

Cancels -D

-f <config file>

Loads the configuration from file.

Mandatory

-D

Starts the process as a daemon.

-G

Sets the effective group ID.

-U

Sets the effective user ID.

Configuring a single level Stick Table Aggregator

The sections below will guide you through aggregating stick tables from several HAProxy Enterprise instances to a new stick table.

Edit the hapee-lb.cfg HAProxy Enterprise configuration files

  1. Replace the HAProxy Enterprise instances' peer lines but the current instance's by a new line corresponding to the Stick Table Aggregator.

    Unaggregated HAProxy Enterprise peers A, B, and C

    HAProxy Enterprise peers A, B, and C's configuration files:

    # /etc/hapee-2.0/hapee-lb.cfg
global
  stats socket /var/run/hapee-2.0/hapee-lb.sock

defaults
  mode http

frontend mytable.local
  bind *:80
  stick-table type ip size 100 expire 1h store http_req_rate(1h) peers mypeers
  http-request track-sc0 src
  http-request deny deny_status 200

peers mypeers
  bind *:54321
  peer A
  peer B
  peer C
    Aggregated HAProxy Enterprise peers A, B, and C

    In HAProxy Enterprise A's configuration file, replace peer lines B and C by peer line AGGR:

    peers mypeers
  bind *:54321
  server A
  server AGGR 192.168.122.13:54321
    In HAProxy Enterprise B's configuration file, replace peer

    lines A and C by peer line AGGR:

    peers mypeers
  bind *:54321
  server B
  server AGGR 192.168.122.13:54321
    In HAProxy Enterprise C's configuration file, replace peer

    lines A and B by peer line AGGR:

    peers mypeers
  bind *:54321
  server C
  server AGGR 192.168.122.13:54321

    Note

    Alternatively, you can define stick tables in the peers section:

    # /etc/hapee-2.0/hapee-lb.cfg
global
  stats socket /var/run/hapee-2.0/hapee-lb.sock

defaults
  mode http

peers mypeers
  bind *:54321
  server A
  server AGGR 192.168.122.13:54321
  table mytable.local type ip size 100 expire 1h store http_req_rate(1h)
  table mytable.aggr type ip size 100 expire 1h store http_req_rate(1h)

  2. Create a new backend with only the very same stick-table line as the one used to create the original, unaggregated stick table.

    The name of the new backend must comply with the naming convention you set up for the aggregate stick table on the from line of the Stick Table Aggregator's configuration file.

    Stick Table Aggregator's from line:

      from .local to .aggr

    Original frontend or backend on HAProxy Enterprise instances:

    frontend mytable.local

    New backend on HAProxy Enterprise instances:

    backend mytable.aggr

    Note

    Since the HAProxy Enterprise instances host both:

    • their original stick table, and,

    • the aggregate stick table,

    you must create a new backend to listen to the aggregate stick table.

    Warning

    Since the new backend section must not generate entries, it

    only contains the stick-table line.

    Unaggregated HAProxy Enterprise peers A, B, and C

    HAProxy Enterprise peers A, B, and C's configuration files contain the following:

    frontend mytable.local
  bind *:80
  stick-table type ip size 100 expire 1h store http_req_rate(1h) peers mypeers
  http-request track-sc0 src
  http-request deny deny_status 200

    Aggregated HAProxy Enterprise peers A, B, and C

    backend mytable.aggr
  stick-table type ip size 100 expire 1h store http_req_rate(1h) peers mypeers

    Full aggregated HAProxy Enterprise peers A's configuration file, with line numbers:

     1 # /etc/hapee-2.0/hapee-lb.cfg
 2 global
 3   stats socket /var/run/hapee-2.0/hapee-lb.sock
 4 
 5 defaults
 6   mode http
 7 
 8 frontend mytable.local
 9   bind *:80
10   stick-table type ip size 100 expire 1h store http_req_rate(1h) peers mypeers
11   http-request track-sc0 src
12   http-request deny deny_status 200
13 
14 backend mytable.aggr
15   stick-table type ip size 100 expire 1h store http_req_rate(1h) peers mypeers
16 
17 peers mypeers
18   bind *:54321
19   server A
20   server AGGR 192.168.122.13:54321

    Note

    The only difference between HAProxy Enterprise peers A, B, and C's configuration files occurs on line 19.

    Warning

    The local and the aggregate stick-table lines must be exactly the same (lines 10 and 15 in the example above).

HAProxy Enterprise peers stop synchronizing their stick tables, and communicate only with the Stick Table Aggregator.

Edit the stktagg.cfg Stick Table Aggregator configuration file

  1. Declare the HAProxy Enterprise instances as peers.

  2. Declare the Stick Table Aggregator itself as a local peer.

  3. Create a naming convention for aggregate stick tables on the from line.

    To aggregate any <string>.local stick tables from HAProxy Enterprise peers A, B and C to stick table <string>.aggr on a Stick Table Aggregator whose name is AGGR:

    #  /etc/hapee-extras/hapee-stktagg.cfg
global
  stats socket /tmp/stktagg.socket

aggregations AGGR
  peer A 192.168.122.222:54321
  peer B 192.168.122.76:54321
  peer C 192.168.122.235:54321
  peer AGGR 192.168.122.13:54321 local
  from .local to .aggr

    Tip

    The default suffix used within the Real-Time Dashboard when referencing aggregate tables is .aggr.

Configuring multilevel Stick Table Aggregators

You can hierarchically organize Stick Table Aggregators in a tree with as many levels as needed.

You can thus aggregate stick tables located in different geographical areas, for example.

Stick Table Aggregators A, B, C, and D are organized as follows:

../../_assets/images/stick-table-aggregator.png

Any <string>.local stick tables hosted by level 0 HAProxy Enterprise peers are aggregated by level 1 Stick Table Aggregators to stick tables <string>.aggr.

Tip

The default suffix used within the Real-Time Dashboard when referencing aggregate tables is .aggr.

Level 0

Level 1

Level 2

HAProxy Enterprise peer

Stick table

Stick Table Aggregator

Aggregate stick table

Stick Table Aggregator

Aggregate stick table

hB.1

mytable.local

B

mytable.aggr

A

mytable.l2

hB.2

mytable.local

hC.1

mytable.local

C

mytable.aggr

hC21

mytable.local

hD.1

mytable.local

D

mytable.aggr

hD.2

mytable.local

Upward and downward peers

The up and down parameters specify whether a peer is upstream or downstream, relative to the current Stick Table Aggregator.

../../_assets/images/upward-and-downward-peers.png

Aggregation from downward peers

A Stick Table Aggregator that receives updates from downward Stick Table Aggregators or HAProxy Enterprise peers forwards the aggregate results to all peers, including the peer that sent the updates.

../../_assets/images/aggregation-from-downward-peers.png

Aggregation from upward peers

A Stick Table Aggregator that receives updates from upward Stick Table Aggregators does not aggregate them and forwards them to its downward peers if they match a forward rule.

../../_assets/images/aggregation-from-upward-peers.png

Redundancy

The group peer line setting creates groups of upward Stick Table Aggregators peers. They are thus seen as a single upward Stick Table Aggregator peer by downward Stick Table Aggregators.

The double link between the level 1 Stick Table Aggregators and the upward, level 0 Stick Table Aggregator peers provides redundancy.

Stick Table Aggregators A, B, AGGR.1, and AGGR.2 are organized as follows:

../../_assets/images/redundancy.png

If Stick Table Aggregator AGGR.1 fails, AGGR.2 continues to aggregate and distribute stick tables.

../../_assets/images/redundancy-failure.png

The table below lists on which peers the various local and aggregate stick tables are created:

Level 0

Level 1

Level 2

HAProxy Enterprise peer

Stick table

Stick Table Aggregator

Aggregate stick table

Stick Table Aggregator

Aggregate stick table

hA.1

mytable.local

A

mytable.aggr

AGGR.1

mytable.l2

hA.2

mytable.local

hB.1

mytable.local

B

mytable.aggr

AGGR.2

mytable.l2

hB.2

mytable.local

The group peer line setting takes only one group identifier string as an argument.

Here are the configuration files of each Stick Table Aggregator in the diagram above.

We list only peers (without socket addresses) and from lines.

Stick Table Aggregator A's configuration file:

peer AGGR.1 <socket address> up group 1
peer AGGR.2 <socket address> up group 1
peer      A <socket address> local
peer   hA.1 <socket address>
peer   hA.2 <socket address>
from any to .aggr

Stick Table Aggregator B's configuration file:

peer AGGR.1 <socket address> up group 1
peer AGGR.2 <socket address> up group 1
peer      B <socket address> local
peer   hB.1 <socket address>
peer   hB.2 <socket address>
from any to .aggr

Stick Table Aggregator AGGR.1's configuration file:

peer      A <socket address> down
peer      B <socket address> down
peer AGGR.1 <socket address> local
from .aggr to .l2

Stick Table Aggregator AGGR.2's configuration file:

peer      A <socket address> down
peer      B <socket address> down
peer AGGR.2 <socket address> local
from .aggr to .l2

Stick Table Aggregator configuration file reference

At startup, the Stick Table Aggregator reads a mandatory text configuration file (/etc/hapee-1.9r1/hapee-stktagg.cfg by default), similar to HAProxy Enterprise configuration files.

It comprises sections made up of one or several lines. Each line is identified by its first keyword and may comprise one or several keywords.

Note

  • In the rest of this document, we often refer to supported sections lines by their first keyword.

  • You can comment out lines by preceding them with a pound sign (#).

  • You can split lines by escaping end of lines with a backslash character (\).

Two sections are supported:

global

Optional. Must be unique, if provided.

aggregations

Mandatory. There may be several aggregations sections in the same configuration file.

global
   [cpu-map set]
   [source IP:port]
   [stats socket IP:port]

aggregations <identifier>
  peer <name> <ip:port> [local | up | down] [group group_id]
  from any to <.suffix>|from <.suffix1,.suffix2,...> to <.suffix> [accept-no-suffix]
  [forward <.suffix1,.suffix2...>]

global section (optional)

The configuration supports only one (optional) global section.

global
   [cpu-map set]
   [source IP:port]
   [stats socket IP:port]

cpu-map (optional)

Same meaning as for HAProxy Enterprise.

Configures the CPU affinity of the Stick Table Aggregator processes.

Comprises one or several CPU sets, each CPU set being a CPU number, or a range of CPU numbers.

CPU sets must be separated by a comma (,).

Select CPUs 0, 5, 6, 7 and 8:

cpu-map 0,5-8

source (optional)

Binds to a specific socket address for outgoing connections.

source lines are declared as follows:

IPv4 addresses
source <ipv4>[:<port>]

Port is optional. Defaults to 0 for random ports.

IPv6 addresses
source <ipv6>:<port>

Port is mandatory if the address is not enclosed by square brackets ([]).

source [<ipv6>][:<port>]

The port is optional if the address is enclosed by square brackets ([]).

stats socket (optional)

Same meaning as for HAProxy Enterprise.

Creates a listener with which you can interact through a TCP or Unix domain socket with the Stick Table Aggregator at runtime.

Connect to 127.0.0.1 on port 11111 to access runtime CLI:

stats socket 127.0.0.1:11111

Use Unix domain socket /tmp/foo to access runtime CLI:

stats socket /tmp/foo

The following commands are available:

show aggrs

Lists all aggregation debugging information.

Gives the list of peers and aggregate tables for all aggregations sections in the configuration file.

The following fields are available:

update

Displays the last key that was updated in the Stick Table Aggregator table.

push

Displays the last key that was pushed to a given peer.

Always the same value as update in normal conditions.

The ack field displays the last key that was acknowledged by a given peer. Usually a bit behind with actual traffic.

state

Protocol state of the peers session with a given peer.

A working peers session shows a state of 0x7.

List aggregations:

$ sudo nc -U /tmp/foo <<< "show aggrs"
aggregations agg_example
  table 'in' aggregated to 'in.aggr' (expire: 600000 nb_entries: 1)
    update: 0x41f03e58 used: 0x41f03e68
    peer 'buster1' (0) state: 0x7 push: 0x41f03e58 ack: 0x41f03a20 sync: 0x41f03a30
    peer 'buster2' (1) state: 0x7 push: 0x41f03e58 ack: 0x41f03a20 sync: 0x41f03a30
    peer 'buster3' (2) state: 0x7 push: 0x41f03e58 ack: 0x41f03a20 sync: 0x41f03a30
    peer 'buster4' (3) state: 0x7 push: 0x41f03e58 ack: 0x41f03a20 sync: 0x41f03a30
show table
show table <aggregation ID> <name>

Dumps the content of a Stick Table Aggregator's aggregate stick table.

For each key, the aggregate entry and the entry sent by each peer are displayed.

Expired entries remain visible, and are reused when the same key is used again.

$ sudo nc -U /tmp/foo <<< "show table agg_example in.aggr"
key=192.168.122.1 http_req_cnt=86747 http_req_rate=(60000)=75040 exp=600000
  buster1: http_req_cnt=62816 http_req_rate=(60000)=51107 exp=533281
  buster2: http_req_cnt=0 http_req_rate=(60000)=0 expired
  buster3: http_req_cnt=0 http_req_rate=(60000)=0 expired
  buster4: http_req_cnt=23931 http_req_rate=(60000)=23931 exp=600000
help

Displays a help message.

aggregations section (mandatory)

An aggregations section declaration keyword must be followed by a mandatory identifier (a string) that is only used internally by the stick table parsers.

Each aggregations section comprises:

  • A list of peers with at least one peer line

  • A unique from line

aggregations <identifier>
  peer <name> <ip:port> [local | up | down] [group group_id]
  from any to <.suffix>|from <.suffix1,.suffix2,...> to <.suffix> [accept-no-suffix]
  [forward <.suffix1,.suffix2...>]

Note

You do not need to configure stick tables on the Stick Table Aggregator since it reads stick table configurations from its remote peer.

peer (mandatory)

peer lines are declared as follows:

peer <peer name> <ip:port> [local | up | down] [group <group_id>]
     [ssl] [crt <path>] [ca-file <path] [verify none | optional | required]

A unique, mandatory peer line with the local keyword identifies the listening peer that listens to incoming remote peer connections.

peer lines without the local keyword identify remote peers.

name (mandatory)

Name of the peer.

ip:port (mandatory)

Socket address of the peer.

up | down (optional)

Optional, exclusive keywords for multilevel settings.

group (optional)

Optional keyword for multilevel settings.

ssl (optional)

Enables TLS/SSL.

crt (optional)

Certificate to present to the remote peer.

ca-file (optional)

File containing the trusted CAs in PEM format.

verify (optional)
none

Verification is disabled.

optional

A client certificate is requested, but verification is performed only if the remote peer provides a certificate in the response.

required

The remote peer certificate is mandatory in the response and verification is always performed.

from (mandatory)

Append a suffix to HAProxy Enterprise peers stick table names to name aggregate stick tables.

First syntax
from any to <.suffix>

Concatenate the names of the stick tables to be aggregated with <.suffix>, ignoring existing suffixes, if any.

Second syntax
from <.suffix1,.suffix2,...> to <.suffix> [accept-no-suffix]

Replaces any suffix found in the list of suffixes by <.suffix>.

<.suffix1,.suffix2,...> is a comma-separated list of suffixes with no spaces.

The accept-no-suffix optional keyword also appends the suffix to stick tables with no suffix.

Stick Table Aggregator stktagg_1 aggregates stick tables mytable hosted by hostA and hostB to mytable.aggr.

global
  cpu-map 2-3

aggregations agg_1
  peer hostA        192.168.122.222:54321
  peer hostB        192.168.122.76:54321
  peer stktagg_1    192.168.122.13:54321   local
  from any to .aggr

If we edit the from line as follows:

from .cpu,.proc to .aggr

the Stick Table Aggregator creates the following aggregate stick tables:

Unaggregated stick table

Aggregated stick table

mytable1.cpu

mytable1.aggr

mytable2.proc

mytable2.aggr

If we edit the from line as follows:

from .cpu,.proc to .aggr accept-no-suffix

the Stick Table Aggregator creates the following aggregate stick tables:

Unaggregated stick table

Aggregated stick table

mytable

mytable.aggr

mytable1.cpu

mytable1.aggr

mytable2.proc

mytable2.aggr

Tip

The default suffix used within the Real-Time Dashboard when referencing aggregate tables is .aggr.

forward (optional)

Only for multilevel settings.

forward <.suffix1,.suffix2...>

The stick table with those suffixes are regarded as forwarded from upward to downward peers.

Note

Updates to these stick tables from a downward peer are ignored.

Aggregation for the middle level:

aggregations agg_1
  peer hostA 192.168.122.222:54321
  peer hostB 192.168.122.76:54321
  peer stktagg 192.168.122.13:54321 local
  peer upstream 192.168.122.235:54321 up
  from any to .aggr
  forward .aggr

Aggregation for the top level:

aggregations agg_1
  peer upstream 192.168.122.235:54321 local
  peer stktagg 192.168.122.13:54321 down
  [...]
  from .aggr to .l2

Tip

The default suffix used within the Real-Time Dashboard when referencing aggregate tables is .aggr.