Stick Table Aggregator
Stick-tables are commonly used to store persistence information by keeping reference information entries about a server where a client connected, such as client IP addresses or client identifiers.
Normally these are tracked by each individual HAProxy process individually, however HAProxy is capable of exporting these entries over a TCP connection via the peers protocol. The stick table aggregator is a daemon which will listen to these announcements from multiple HAProxy nodes and export a copy of the stick table to each with the combined values. This will be primarily of use in an active-active HAProxy cluster (such as BGP) where a given client may have requests distributed through multiple HAProxy servers, though it can be useful in other setups where consistency is important.
This combined stick-table can then be referenced by any of the HAProxy servers in the cluster in order to make decisions about scraping, DDoS attacks, and other risks.
The stick table aggregator was added with HAProxy Enterprise 1.7r2, but will work with HAProxy Enterprise 1.7r1 clients. If used with HAProxy Enterprise 1.7r1 clients stick table entries might not expire at the proper time if HAProxy Enterprise is reloaded.
- stick-table source
A stick-table whose entries must be aggregated to others.
- stick-table destination
A stick-table whose entries are the results of an aggregation.
Install the stick-table aggregator
As with other HAProxy Enterprise modules, you must install the HAProxy Enterprise stick-table aggregator using the Linux distribution packaging manager.
The stick-table aggregator depends on the libev4 package on Debian based systems and libev package on RHEL based systems. This package does not exist on RHEL 6. Hence, the stick-table aggregator is not supported for RHEL systems prior to RHEL 7.
The stick-table aggregator needs to be run on a system which isn't running an HAProxy instance that needs to be a part of the cluster (e.g. doesn't have stick tables which need to be aggregated by this daemon); this is due to a technical limitation of the peers protocol where announcements from the same IP address will be ignored (and which we hope to resolve in a future release).
Install the stick-table aggregator on Debian based systems
Enter this command:
$ sudo apt-get install hapee-extras-stktagg
Manage the Stick Table Aggregator process
You can start, stop, or restart the stick-table aggregator using the System V init script
/etc/init.d/hapee-1.7r2-stktagg or using systemd as follows:
$ sudo service <start|stop|restart|status> hapee-extras-stktagg
By default the stick table aggregator will start as root; to change this behavior, you can modify the file
/etc/default/hapee-1.7r2-stktagg/ file to start it with hapee-stktagg as a user and hapee as group as follows:
STKTAGG_OPTIONS="-D -U hapee-stktagg -G hapee"
Configure the stick-table aggregator
The following table lists the command line options:
Display help message
Configuration filename (mandatory)
Start the aggregator as a daemon
Group effective ID
User effective ID
Allows the user to provide a source IP address
Allows the user to configure a socket to which to connect in order to dump data
:~# echo "show table aggr_c3811-anycast src_table.aggr" |socat tcp-connect:127.0.0.1:4545 -|head key=192.168.1.123 gpt0=0 gpc0=0 gpc0_rate=(10000)=0 conn_rate=(10000)=0 http_req_cnt=0 http_req_rate=(10000)=0 http_err_cnt=0 http_err_rate=(10000)=0 gpc1=0 gpc1_rate=(10000)=0 key=192.168.1.124 gpt0=0 gpc0=0 gpc0_rate=(10000)=0 conn_rate=(10000)=0 http_req_cnt=0 http_req_rate=(10000)=0 http_err_cnt=0 http_err_rate=(10000)=0 gpc1=0 gpc1_rate=(10000)=0
You integrate an HAProxy peer with the aggregator by adding the aggregator peer information to the peers section of the HAProxy configuration, define a stick table which the node will use, and a second stick table to hold the received aggregated values.
HAProxy peers A, B, and C are already members of a cluster (they don't need to be, but makes for an explanation easier to understand if you are familiar with the peers protocol).
Their configuration files contain the following peers (IP/Port information is replaced by '...' in these examples):
peers foo.peers peer A ... peer B ... peer C ...
To aggregate their stick-tables with an aggregator peer named AGGR in A, add the following to peer A's configuration file:
peers foo.peers peer A ... peer AGGR ...
Add the following to peer A's configuration file:
peers foo.peers peer B ... peer AGGR ...
On the AGGR aggregator side, the configuration file would contain in the aggregations section the following peer lines:
peer A ... peer B ... peer C ... peer AGGR ... local
Peer A, B, and C would stop synchronizing their stick-tables and would communicate only with the AGGR aggregator peer.
Declare an additional stick table (on top of the one already being used) with the name specified in the from line of the aggregator. It should be identical to the current stick table except for its name (and nothing will be using it except to read the aggregated values)
Example: If you declare the line from .cpu to .aggr in the AGGR configuration file, you must also declare the following backends in the configuration files of HAProxy peer A, B, and C with exactly the same stick-table line in both foo.cpu and foo.aggr backend sections:
backend foo.cpu stick-table type ... backend foo.aggr stick-table type ...
The configuration file
The configuration file for the stick-table aggregator is located at
/etc/hapee-1.7r2/hapee-stktagg.cfg and is required for it to operate.
This file is similar to other HAProxy configuration files containing sections made up of several lines:
Each line is identified by its first keyword and may be made of one or several keywords. In this document we reference sections/lines by the first keyword used (e.g. 'peer').
Lines may be commented out using the hash character (#).
The end of the line can be exited using a backslash character (\) to split a line into several.
Currently, only two sections are supported: global and aggregations. The configuration file only has one global section but may contain several aggregations sections.
This section currently supports only one line: cpu-map. It has the same meaning as cpu-map setting for HAProxy and is used to configure the CPU affinity of the stick-table aggregator process(es). It is made of one or several CPU sets, each CPU set being a CPU number, or a range of CPU numbers. These CPU sets must be separated by a comma character (,).
# Select 0 and from 5 up to 8 CPUs. cpu-map 0,5-8
An aggregations section must follow the rules below:
The aggregations keyword (starting the section) must be followed by an identifier (e.g. aggregations example_com). This name is an internal identifier and is only used for identification and referencing inside this configuration file.
Supports only two lines keywords:
Has one or more peer lines (HAProxy servers it will listen to stick-table activity from)
Has one from line (telling it what the name of the new aggregated table is)
You declare the peer lines as follows:
peer <name> <ip:port> [local | up | down] [group group_id]
The peer lines contain two mandatory settings: their name and IP/port, followed by:
optional exclusive keywords:
The IP of the server running the stick table aggregator, exactly one of these lines is required in each aggregator section.
Only used in multi-level setups, identifies a peer which is higher in the multilevel tree than the current aggregator and will send aggregated data from this aggregator to other aggregators.
Only used in multi-level setups, identifies a peer which is lower in the multilevel tree than the current aggregator and will forward any updates it receives to all its peers.
optional group setting.
Among these peers, one must be local and followed by the local keyword, accompanied by one IP address that identifies the listening peer accepting incoming peer connections.
Other peer lines without the local optional keywords serve to identify remote peers (which will send and receive stick table values).
Among the remote peers, there may be other stick-table aggregators that must be identified by an up or down keyword.
The objective of the from line is to instruct the stick-table aggregator on how to name the stick-table destinations, or aggregated stick-tables.
The from keyword accepts two forms of syntax:
The first form of this syntax asks the stick-table aggregator to build the aggregated stick-table names by concatenating the names of the stick-tables to be aggregated with <suffix> as the suffix without regard to pre-existing suffixes.
from any to <suffix>
The second form of this syntax asks the stick-table aggregator to build the aggregated stick-table names from the names of stick-tables to be aggregated and to replace any suffix found in the list of suffixes with <suffix> suffix.
from <suffix1,suffix2,...> to <suffix> [accept-no-suffix]
You can use the optional keyword accept-no-suffix to accept any name of stick-tables to be aggregated without any suffix found in the <suffix1,suffix2,...> list. In this case, the names of the aggregated stick-table are built by concatenating the names of the stick-tables to be aggregated with the <suffix> suffix.
Note: a suffix string must have a period (.) as the first character. The <suffix1,suffix2,...> must be made of suffixes separated by a comma character (,), without any space between each suffix.
This following complete configuration file example sets up a stick-table aggregator that:
Has stktagg_1 as peer name
Aggregates any stick-table hosted by hostA and hostB peers with another stick-table that must also be hosted by theqse two peers.
If foo is the name of the stick-table to be aggregated, foo.aggr would be the name of the aggregated stick-table.
Each of the non-local peers (hostA and hostB) are running HAProxy with a peers section with two entries: the specified IP+Port and the stick table aggregator IP+Port
The HAProxy nodes must have stick tables defined which they will send events for and another which will hold the aggregated values
global cpu-map 2-3 aggregations agg_1 peer hostA 192.168.0.200:10000 peer hostB 192.168.0.201:10001 peer stktagg_1 192.168.0.1:9000 local #IP of the host running this configuration file from any to .aggr
If we replace from... line above with this one:
... from .cpu,.proc to .aggr
the stick-table aggregator would aggregate foo1.cpu (resp. foo2.proc) to foo1.aggr (resp. foo2.aggr) stick-table. With an additional accept-no-suffix keyword, foo stick-table could also be aggregated to foo.aggr stick-table.
Example 2: hapee-lb.cfg
peers aggr peer haproxy1 192.168.1.10:10000 peer stkaggr1 192.168.1.20:10000 frontend web bind *:80 http-request track-sc0 src table src_table default_backend web backend src_table stick-table type ip size 1m expire 10m store conn_rate(10s),http_req_cnt,http_err_cnt,http_req_rate(10s),http_err_rate(10s) peers aggr backend src_table.aggr stick-table type ip size 1m expire 10m store conn_rate(10s),http_req_cnt,http_err_cnt,http_req_rate(10s),http_err_rate(10s) peers aggr
Example 3: /etc/hapee-extras/hapee-stktagg.cfg
global cpu-map 1 source 192.168.1.20:0 stats socket 127.0.0.1:4545 aggregations agg_example # HAProxy peers peer haproxy1 192.168.1.10:10000 peer haproxy2 192.168.1.11:10000 peer haproxy3 192.168.1.12:10000
Multi-level stick-table aggregators
Stick-table aggregators may be cascaded and organized hierarchically in a tree. This can be useful to aggregate stick-tables located in different places (datacenters).
Two rules apply when forwarding aggregation results between aggregator peers:
A peer aggregator that receives updates from downward peers (aggregators or not) forwards the aggregation results to all its peers without any exception, even to the peer that sent the updates.
A peer aggregator which receives updates from upward aggregators aggregates them and forwards the results to its downward peers.
For example with such an organization of four A, B, C, and D aggregators:
A Level 0: for .aggr to .cluster _/|\_ _/ | \_ _/ | \_ _/ | \_ _/ | \_ / | \_ B C D Level 1: for any to .aggr / \ / \ / \ / \ / \ / \ hB.1 hB.2 hC.1 hC.2 hD.1 hD.2 Level 2: (HAProxy peers)
Any xxx stick-table hosted by hB.1 & 2 (resp. hC.1 & 2 and hD.1 & 2) HAProxy peers would be aggregated by B (resp. C and D) with xxx.aggr as the aggregated stick-table.
At level 0 in the tree, peer A aggregator is seen as another peer from level 1 aggregators; hence it would receive the aggregation results of level 2 downward HAProxy peer xxx stick-tables, presented as xxx.aggr stick-table entries.
These aggregations results would also be aggregated again to stick-table xxx.cluster and redistributed to level 1 B, C and D peers using the setting from .aggr to .cluster.
Consequently, all level 2 peers would receive their own copy of xxx.cluster stick-table contents which would be the aggregation results of B, C and D xxx.aggr stick-tables.
Upward aggregator peers
There is another multilevel feature which can add a redundancy service to the aggregation services provided by aggregators. You can group upward aggregator peers to be viewed as a unique upward aggregator peer from its downward aggregators peers using the line setting group on the peer line.
This configuration looks like this:
AGGR.1 AGGR.2 Level 0: from .aggr to .cluster / \ / \ / \ / \ / X \ / _/ \_ \ / _/ \_ \ /___/ \___\ A B Level 1: from any to .aggr / \ / \ / \ / \ hA.1 hA.2 hB.1 hB.2 Level 2: (HAProxy peers)
In this double tree organization of stick-table aggregators, if you remove AGGR.1, the remaining tree is similar to the previous tree, but with only two branches instead of three.
Similarly, if you remove AGGR.2 from this tree, you also get two branches instead of 3.
Hence, AGGR.1 & 2 aggregators offer the same aggregations as the previous A aggregator: they aggregate xxx.aggr stick-tables of A and B, respectively, which are themselves aggregations results of hA.1 & 2 and hB.1 & 2 xxx stick-tables, respectively.
The double link between the level 1 aggregator and their two upward level 0 aggregator peers offers a redundancy service. If one of the level 0 aggregators fails, the remaining one continue its aggregation services.
The group peer line setting takes only one string as argument: a group identifier.
The following is an example the configuration files of each aggregator present in the double tree above. We list only the peer (without IP/port) and from lines.
A configuration file:
# A configuration file peer AGGR.1 ... up group 1 peer AGGR.2 ... up group 1 peer A ... local peer hA.1 ... peer hA.2 ... from any to .aggr
B configuration file:
# B configuration file: peer AGGR.1 ... up group 1 peer AGGR.2 ... up group 1 peer B ... local peer hB.1 ... peer hB.2 ... from any to .aggr
AGGR.1 configuration file:
peer A ... down peer B ... down peer AGGR.1 ... local from .aggr to .cluster
AGGR.2 configuration file:
peer A ... down peer B ... down peer AGGR.2 ... local from .aggr to .cluster