Extend HAProxy With Stream Processing Offload Engine (SPOE)

The HAProxy Stream Processing Offload Engine filter enables you to extend HAProxy in any language without modifying its core codebase.

Imagine you’re living in the distant future and you’re stomping around town in the latest 15-foot tall mech robot. You’ve outfitted it in the latest gadgetry. It’s fast, agile and responsive. When the new arm cannon upgrade becomes available next month, you’ll be able to augment your robot by simply heading to the nearest shop and having the cannons snapped into place.

Although this is straight out of a sci-fi movie, there are some clear parallels to the work we’ve been doing at HAProxy Technologies. HAProxy is the world’s fastest and most widely used software load balancer. Its popularity is due to its high performance, reliability, and flexible configuration language. Now, we want to provide the ability to extend it using any programming language without the complexity of modifying its core codebase. Want to add traffic shadowing capabilities? Or maybe you’d like to route real-time data to a Machine Learning system that can perform some advanced analysis?

In order to expand the possibilities of what HAProxy can do, the Stream Processing Offload Engine (SPOE) was invented. It provides a way to send live traffic to external programs that can process the data out-of-band, not unlike snapping an upgrade onto your mech robot. In this blog post, you’ll be taken on a tour of the SPOE and, we hope, become inspired by your own futuristic possibilities.

A Bit of History

The SPOE has been increasingly refined over the past few versions. It was added as a feature in HAProxy 1.7. It matured in HAProxy 1.8 and 1.9. During upcoming versions, you’ll see it further improved and enriched.

In HAProxy 1.7, the concept of filters was introduced. Filters provide hooks into specific phases of the load balancer’s traffic processing. They allow HAProxy to be extended with code that can intercept real-time, streaming data at various points during the request/response lifecycle. Filters are written using the C programming language and compiled into HAProxy.

In version 1.7, HAProxy Technologies put it to use by rewriting the HTTP compression logic as a filter. In 1.8, an HTTP cache for small objects was added. Its storage is built on the stream processing capabilities of a filter. During that version, we also created a new kind of filter called an SPOE filter. The idea was to make extending HAProxy easier and more accessible, with the specific aim of allowing you to do so with any programming language. An SPOE filter forwards streaming data to an external program.

There are plenty of reasons why you might want to stream load balancer data to an external program. You might not want to get involved in developing HAProxy’s own source code. Or, you might do it to take advantage of a particular programming language’s libraries and features. Or, you may need to develop a proprietary extension that you’re unable to open source.

In the next few sections, we’ll introduce you to the SPOE filter, explain its architecture, and give you a high-level overview of how it works.

Stream Processing Setup

To understand how an SPOE filter works, you should become familiar with three terms:

  • Stream Processing Offload Engine (SPOE)

  • Stream Processing Offload Agent (SPOA)

  • Stream Processing Offload Protocol (SPOP)

The Stream Processing Offload Engine is a type of filter that gives you access to real-time data from HAProxy. It allows work to be offloaded to self-hosted components. A component that does the work is called a Stream Processing Offload Agent. It receives data from an SPOE filter. Data is exchanged between the filter and the agent via a new, binary protocol over TCP called the Stream Processing Offload Protocol.

The following diagram shows how an SPOE interacts with an SPOA via the SPOP. For the rest of this article, I’ll use these acronyms sparingly and stick to the terms: engine, agent and protocol.

spoe sending data

An SPOE sends data to an SPOA via the SPOP

HAProxy configuration

Creating an engine begins with adding a filter directive to your HAProxy configuration file. This turns the stream processing on for a particular proxy (e.g. frontend or listen) or backend section. You can give the filter a name to differentiate it from other engines, although this is not mandatory.

frontend myproxy
filter spoe [engine <name>] config <spoe-config-file>

Giving it a name allows you to configure several engines that points to the same SPOE configuration file. Otherwise, the configuration file will be dedicated to a single engine. The filter directive links to a separate SPOE config file that specifies how your engine will communicate with the agents and which load balancer events to hook into. You’ll also need to add a backend section containing the agent servers.

filter points to separate file

A filter points to a separate config file

Here’s an example that uses a filter directive to stream data to a fictitious service that performs IP reputation scoring on every new client connection.

frontend myproxy
mode http
bind :80
# Declare filter and its config file
filter spoe engine ip-reputation config iprep.conf
# Reject connection if the IP reputation is under 20
tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 }
default_backend webservers
# Normal web servers backend
backend webservers
mode http
balance roundrobin
server web1 192.168.0.5:80 check
# Backend used by the ip-reputation SPOE
backend agents
mode tcp
balance roundrobin
timeout connect 5s # greater than hello timeout
timeout server 3m # greater than idle timeout
option spop-check
server agent1 192.168.1.10:12345 check
server agent2 192.168.1.11:12345 check

In this example, the ip-reputation engine will watch traffic coming into the myproxy frontend. The iprep.conf file configures where to send the data. Ultimately, it’s forwarded to an agent in the agents backend, which does a calculation and sets a variable named ip_score. The variable is then used in a tcp-request content reject rule to deny clients that have an IP reputation score lower than 20.

HAProxy attaches the prefix iprep to variables that the agents set. Prefixes are used to avoid name clashes between engines and are defined in the SPOE configuration file. Currently, a large part of an agent’s power comes from its ability to feed data back to HAProxy through the use of variables.

SPOE configuration

Here’s how iprep.conf looks:

[ip-reputation]
spoe-agent iprep-agent
messages check-client-ip
option var-prefix iprep
timeout hello 2s
timeout idle 2m
timeout processing 10ms
use-backend agents
log global
spoe-message check-client-ip
args ip=src
event on-client-session if ! { src -f /etc/haproxy/whitelist.lst }

The full SPOE configuration syntax is documented within the HAProxy git repository, but let’s go over the highlights. At the top of the file is a scope enclosed in square brackets. A scope lets you tell one engine from another when they’re both configured in the same file. It matches the filter engine name that you set in the HAProxy configuration file. Each named engine gets its own scope like this. If you didn’t assign a name, then you wouldn’t need the scope declaration.

The spoe-agent line begins a part of the file that defines how the engine communicates with its agents. The timeout lines indicate how long HAProxy will wait for an agent to respond during the initial handshake when the connection has been idle, and when processing a message. The use-backend directive tells the engine which backend in the HAProxy configuration contains the agents. That’s where the stream of events will be sent.

HAProxy does not send all of the available information about a connection or request to an agent. In many cases, that would lead to information overload. Instead, you have to decide which pieces of data are important to you. The spoe-agent section sets a list of space-delimited message types via a messages directive. The message types are defined in separate spoe-message sections. Inside each, an args line lists the message’s arguments, which map back to HAProxy fetch methods (e.g. src). Multiple arguments are separated by spaces. It’s okay to pass arguments that don’t have a key (e.g. src instead of ip=src).

The event line tells the engine when to intercept data and send a message. The following events are used to trigger sending messages:

Event

Meaning

on-client-session

Triggered when a new client session is created. This event is only available for SPOE filters declared in a frontend or a listen section.

on-frontend-tcp-request

Triggered just before the evaluation of tcp-request content rules in a frontend. This event is only available for SPOE filters declared in a frontend or a listen section.

on-backend-tcp-request

Triggered just before the evaluation of tcp-request content rules in a backend. This event is skipped for SPOE filters declared in a listen section.

on-frontend-http-request

Triggered just before the evaluation of http-request rules in a frontend. This event is only available for SPOE filters declared in a frontend or a listen section.

on-backend-http-request

Triggered just before the evaluation of http-request rules in a backend. This event is skipped for SPOE filters declared in a listen section.

on-server-sessions

Triggered when a session with the server is established.

on-tcp-response

Triggered just before the evaluation of tcp-response content rules.

on-http-response

Triggered just before the evaluation of http-response rules.

 You can, as in this example, append an ACL if statement to control when to, or when not to, invoke sending the message. Here, we’ve whitelisted some IP addresses for which we don’t want to run the IP reputation service. Without an ACL, a message is sent every time the event fires.

Did you know?

Another way to trigger sending messages to agents is by using the http-request send-spoe-group directive. This requires you to set up a group of messages to send via a spoe-group section in your SPOE configuration file.

After an agent processes a message, it can send back a list of actions that HAProxy should apply. Currently, you can tell HAProxy to set or unset a variable. Future versions will add many other kinds of actions.

Agent code

A Stream Processing Offload Agent can be written in any language and would listen on a TCP port and speak to HAProxy using the Stream Processing Offload Protocol. The provided example IP reputation service is written in C and randomly assigns a score to each IP address and returns a variable named ip_score. We won’t go into detail about the code. Instead, we’ll give a high-level overview of the protocol. However, you can check out the following libraries too:

The Stream Processing Offload Protocol

The communication between HAProxy and the agents happens over a binary protocol called the Stream Processing Offload Protocol (SPOP). You can read the official documentation on the HAProxy project’s website.

The basic unit of the protocol is a frame. Frames package up data, with each frame being a certain type and serving a particular purpose. Frames are prefixed with four bytes that tell the total frame length. That’s followed by a single byte that identifies the frame type. Next, there is some metadata, which contains flags, a stream ID, and a frame ID. Then comes the frame’s payload.

frame in spop

A frame in the Stream Processing Offload Protocol

Here is the list of currently supported frame types:

Type (ID)

Description

0 – UNSET

Used for all frames except the first when a payload is fragmented.

1 – HAPROXY-HELLO

Sent by HAProxy when it opens a connection to an agent.

2 – HAPROXY-DISCONNECT

Sent by HAProxy when it wants to close the connection.

3 – NOTIFY

Sent by HAProxy to pass information to an agent.

101 – AGENT-HELLO

Reply to a HAPROXY-HELLO frame, when the connection is established.

102 – AGENT-DISCONNECT

Sent by an agent just before closing the connection.

103 – ACK

Sent to acknowledge a NOTIFY frame.

 

A connection between HAProxy and an agent is established when HAProxy sends a HAPROXY-HELLO frame and the agent responds with an AGENT-HELLO. During this handshake, the protocol version to use and the parameters of the transaction (e.g. maximum frame size and supported capabilities) are communicated within the payload. At this stage, the stream ID and frame ID should both be set to zero.

connection initialized with a handshake

A connection is initialized with a handshake

If for any reason, an incompatibility is detected during the handshake, HAProxy will send a HAPROXY-DISCONNECT frame. This frame contains a status code that the agent can use to diagnose what went wrong. The agent is responsible for then replying with an AGENT-DISCONNECT frame and closing the connection. It can also initiate a disconnect from its end. In that case, it simply closes the connection and doesn’t need to wait for a response from HAProxy.

error causes a disconnect

An error causes a disconnect

If both sides agree to proceed, the connection is considered established and ready to use. At that point, HAProxy can begin sending NOTIFY frames that contain SPOE messages, as set up in the SPOE configuration’s messages list. These frames have stream IDs and frame IDs. When the agent code parses the frames, it looks for messages with names that it knows how to process (e.g. check-client-ip). It acknowledges each frame with an ACK frame, which includes a matching stream ID. The payload of an ACK may list actions for HAProxy to perform.

sending notify frames

Notify frames are responded to with Acks

Frames cannot exceed the maximum size that was negotiated during the initial handshake. However, it is possible to divvy large payloads up as fragments. This capability is announced during the handshake. A fragmented message is sent as frames that have the same stream ID and frame ID. Its last frame adds a FIN flag.

The payload of a NOTIFY frame is perhaps the most interesting because it contains the real-time traffic information that you’re streaming to the agent. For each spoe-message section that you’ve added to your SPOE configuration file, a message in a NOTIFY frame will be sent when the corresponding event is triggered. Also, if you’ve attached multiple types of messages to the same event, they’ll all be sent.

For example, in iprep.conf, we had defined a message named check-client-ip:

spoe-message check-client-ip
args ip=src
event on-client-session if ! { src -f /etc/haproxy/whitelist.lst }

When the on-client-session event is triggered, which happens when a client first connects to HAProxy (and, in this case, if the client is not whitelisted), then HAProxy sends a NOTIFY frame that contains the message’s name and arguments in its payload. So, in this example, the name check-client-ip along with the client’s source IP, which is recorded via the src fetch method, are included in the message.

Actions

After an agent has finished processing the data that HAProxy has sent to it, you have the option of instructing the load balancer to perform an action. At this time, you’re able to either set or unset a variable in HAProxy. You can then add ACL statements to your HAProxy configuration file that check the variable and later perform an operation, such as choosing a backend or denying the request.

Our fictitious IP reputation service checks the client’s IP address and, based on that, sets a variable that contains their score. We then use that variable to decide whether to allow or deny the connection. Actions are included in ACK frames. The payload of an ACK can contain a list of actions, such as set-var. When using set-var and unset-var, you’ll assign a scope (e.g. session) to the variable to control the phase of the request/response lifecycle in which the variable will live.

In our previous example, the HAProxy configuration reads the sess.iprep.ip_score variable that was set by the agent.

tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 }

If its value is less than 20, the connection is rejected. You can also log variables, like this:

http-request capture var(sess.iprep.ip_score) len 3

Then, the variable will be written to the access log. You’ll find it between curly braces, as in this example where the IP reputation score is 92:

192.168.112.1:43670 [19/Feb/2019:21:40:41.197] myproxy
webservers/web1 0/0/0/0/+0 200 +101 - - ----
1/1/1/1/0 0/0 {92} "GET / HTTP/1.1"

Or you might set them as response headers and view them in your browser.

SPOP Connections Management

HAProxy can reuse existing, open connections to an agent. By default, once the connection is established, it is locked during the time of a NOTIFY ACK exchange. So, only one stream uses it at any particular time. However, it’s marked as idle once the ACK frame is received. Then, other streams can use it.

This functionality is similar to how HTTP Keep-Alive works when HAProxy connects to a backend server. It’s the easiest way to handle reusing an open connection, but there’s a way to get better performance: During the protocol handshake, support for pipelining can be negotiated. Pipelining is the ability for a peer to decouple NOTIFY and ACK frames. Or, in other words, many streams will be able to share the same connection simultaneously. It works much like HTTP/2 multiplexing.

It’s also possible to enable asynchronous mode. In this state, any live connection can be used to transport frames for the same message. So, a NOTIFY frame could go out on one connection and the ACK could come back over a different connection, which offers performance benefits. See the official documentation for more information.

Conclusion

In this blog post, you learned about the Stream Processing Offload Engine, its protocol, the Stream Processing Offload Protocol, and the Stream Processing Offload Agents that receive and process messages from HAProxy. The agents can also instruct HAProxy on actions to perform. With these tools, you’re able to extend HAProxy using any programming language by hooking into events that fire during a session. This allows you to stream real-time data to applications that augment your load balancer with extra functionality. In upcoming posts, we’ll dig into some examples of using an SPOE filter to add new functionality to HAProxy, including the implementation of traffic shadowing.

HAProxy Enterprise includes a robust and cutting-edge codebase, an enterprise suite of add-ons, expert support, and professional services. It even includes a single-sign-on module, which uses an SPOE filter, that authenticates your clients with an LDAP server. Contact us to learn more or get your HAProxy Enterprise free trial today!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.