Announcing HAProxy Data Plane API 2.4
Version Update

HAProxy Data Plane API 2.8 is now the latest version. Learn more

Version 2.4 improves its e2e tests, revamps how logging in the HAProxy Data Plane API works, adds support for namespace filtering in Consul Service Discovery, improves runtime capabilities for maps and ACLs, adds server-template support and adds log_targets to global and defaults sections.

HAProxy Data Plane API version 2.4 continues efforts from the previous version to reduce complexity for users and add freedom to configure the Data Plane API logs to suit your environment, with an easy-to-understand API configuration. Our main goal is to fully support all of HAProxy’s configuration options, which we’ve made progress on by adding log_targets to global and default sections. Also, while the previous release added support for the resolvers section, this version adds support for the server-template line, which works with resolvers to enable DNS-based service discovery and scaling.

To help make using the API less complex, version 2.4 introduces the ability to add multiple map and ACL file entries as a single atomic command, if used with HAProxy 2.4 or greater. It leverages the prepare, add and commit commands in the HAProxy Runtime API.

Contributors also worked to bring the overall quality of the Data Plane API to a higher level, and that can be seen in a big overhaul of the project’s end-to-end tests and some tweaks to its continuous-delivery processes.

All of these changes improve usability and provide more freedom in configuring logging, while still continuing to progress closer to covering the full feature set of HAProxy. This wouldn’t be possible without the great effort of our community with some great contributions in code, bug reports and feature requests!

Logging Rework

Before this version, you would configure logging with the --log CLI options or similar directives in the API’s configuration file. These methods let you record the API’s log, but only to a single location. It looked something like this:

$ sudo dataplaneapi \
--host 127.0.0.1 \
--port 5555 \
--haproxy-bin /usr/sbin/haproxy \
--config-file /etc/haproxy/haproxy.cfg \
--reload-delay 5 \
--reload-cmd "service haproxy reload" \
--restart-cmd "service haproxy restart" \
--userlist haproxy-dataplaneapi \
--transaction-dir /tmp/haproxy
--log-to file
--log-file /var/log/dataplaneapi/dataplaneapi.log
--log-level info
--log-format json

That could be a little cumbersome, and not to mention that if you wanted to forward those logs to somewhere else you had to work your way around this on your own. Starting from 2.4, you have the ability to configure multiple log targets for the Data Plane API in the configuration file. For example, this configuration matches the previous CLI options:

— yaml—

log_targets:
- log_to: file
log_file: /var/log/dataplaneapi/dataplaneapi.log
log_level: info
log_format: json
log_types:
- access
- app

—hcl—

log_targets = [
{
log_to = "file"
log_file = “/var/log/dataplaneapi/dataplaneapi.log”
log_level = "info"
log_format = "json"
log_types = [
"access",
"app",
]
},
]

Suppose you want to configure multiple log targets. Let’s say you want to see all of your debug-level logs on the terminal’s stdout, store info-level application logs in a JSON file, and send Apache formatted access logs to a dedicated syslog instance. You could do something like this:

—yaml—

log_targets:
- log_to: stdout
log_level: debug
log_format: text
log_types:
- access
- app
- log_to: file
log_file: /var/log/dataplanepi.log
log_level: info
log_format: json
log_types:
- app
- log_to: syslog
log_level: info
syslog_address: 127.0.0.1
syslog_protocol: tcp
syslog_tag: dataplaneapi
syslog_level: debug
syslog_facility: local0
log_types:
- access

—hcl—

log_targets = [
{
log_to = "stdout"
log_level = "debug"
log_format = "text"
log_types = [
"access",
"app",
]
},
{
log_to = "file"
log_file = "/var/log/dataplanepi.log"
log_level = "info"
log_format = "json"
log_types = ["app"]
},
{
log_to = "syslog"
log_level = "info"
syslog_address = "127.0.0.1"
syslog_protocol = "tcp"
syslog_tag = "dataplaneapi"
syslog_level = "debug"
syslog_facillity = "local0"
log_types = ["access"]
},
]

See our HCL formatted example and YAML formatted example to learn how to configure logging to suit your needs.

What also changed is that there is now a distinction between the two types of log sources: application and access logs. Access logs were recorded on two lines that were a bit hard to follow when multiple API calls were made, one line representing the start of a request and one for completing it:

time="2021-10-01T12:51:57+02:00" level=info msg="started handling request" method=GET remote="[::1]:36924" request=/v2
time="2021-10-01T12:51:57+02:00" level=info msg="completed handling request" length=705B request_id=01FGXQPVAYE48Z26JQB81D99JJ status=200 took=4.468977ms

These access logs are now replaced with a single Apache Common Log formatted log line:

time="2021-10-01T12:54:44+02:00" level=info msg="[::1] - - [01/Oct/2021:12:54:44 +0200] \"GET /v2 HTTP/1.1 200 705 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36\" 4768"

The default format uses the following HAProxy log variables: %h %l %u %t "%r %>s %b "%{Referer}i" "%{User-agent}i" %{us}T. However, you can customize it using the acl_format setting to better suit your needs.

All other logs, such as events recorded when the API starts, are considered application logs, and with the log_type you can define what type of logs you want to send to the configured log target.

Deprecation warning

The CLI options for logging and the older log section in the configuration file remain in place, but they are deprecated, meaning that in version 2.5 they will be removed. Currently, they are only used when no log_targets section is configured in the configuration file to retain users’ environments and expected behavior. Once there are log_targets configured, only those are used. The CLI options and old log section is ignored in that case.

Extended HAProxy Configuration Keywords Support

The HAProxy Data Plane API is continuously improving its coverage of supported HAProxy keywords with the goal of becoming simpler and easier to use. In that area, 2.4 brings a couple of improvements.

Support for server-templates

We’ve added a new API resource, /v2/services/haproxy/configuration/server_templates, to support the server-template keyword in the HAProxy configuration backend sections. This is a feature that has been requested multiple times and finally lands in the HAProxy Data Plane API!

Global and defaults log_targets

Previously, you could add /v2/services/haproxy/configuration/log_targets only to backends and frontends sections, which you specified using parent_type and parent_name query string parameters. With 2.4 you can add them to global or default sections too using parent_type=global or parent_type=defaults query string parameters. Note that you don’t need to specify parent_name in this case.

Minor changes

Version 2.4 adds several fields to the following resources:

/v2/services/haproxy/configuration/global:

  • Added hard-stop-after

  • Added localpeer

  • Added server_state_base

  • Added server_state_file

/v2/services/haproxy/configuration/defaults:

  • Added client_fin_timeout

  • Added server_fin_timeout

/v2/services/haproxy/configuration/backends:

  • Added uri_path_only to balance field

Extended HAProxy Runtime API Commands Support

In the HAProxy 2.4 Runtime API there is an ability to add multiple maps and ACL file entries in an atomic way using a prepare, add and commit workflow. This feature allows the Data Plane API to add multiple entries to map_file and acl_file resources in one API call atomically without worrying about partial data inserts in cases of failure.

You can do that by using the PUT method on  /v2/services/haproxy/runtime/acl_file_entries  and /v2/services/haproxy/runtime/maps/{name} resources, respectively.

Minor Quality of Life Improvements

HAProxy Data Plane API 2.4 comes out with a batch of quality-of-life improvements for users and contributing developers. It includes a series of bug fixes that have been backported to v2.3.6 already, the Consul Service Discovery now has the ability to filter services by namespace, and the Configuration-Version header is now returned even in the case of an error response.

To get to higher quality code and attract more contributors, the Data Plane API now has a more readable end-to-end testing suite, which will make first-time contributions easier and safer.

Contributors

Contributor

Area

Amel Husic

FEATURE  CLEANUP  BUG  BUILD

Dario Tranchitella

FEATURE  BUG  TEST

Georgi Dimitrov

BUG

Goran Galinec

FEATURE  CLEANUP  DOC

Marko Juraga

FEATURE  BUG  BUILD  DOC  REORG  TEST

Piotr Olchawa

BUG

Robert Maticevic

FEATURE

Zlatko Bratkovic

FEATURE  BUG  BUILD  CLEANUP

Conclusion

The HAProxy Data Plane API version 2.4 focuses on quality-of-life improvements and general usability. With the new logging system, you can better monitor what is happening with the API, and easily include it in your logging systems. It further expands to cover more features from both the configuration file and the Runtime API, and shows an effort to be more maintainable and easier to contribute to with the revamped end-to-end testing suite. The API continues to get better, thanks to the active community surrounding it. If you’d like to get involved, check out our GitHub repository to learn more!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.