In HAProxy 2.1, you will see significant performance gains in key areas, exciting new features including Dynamic SSL Certificate Updates, FastCGI, and Native Protocol Tracing, and a streamlined codebase.
HAProxy 2.1 is a technical release that is focused on improving the core codebase and developing better debugging and observability tools for contributors, support engineers, and developers. You will see a significant performance increase in multiple areas. This is due to optimizations such as scheduler improvements for dealing with high connection rates and increased efficiency from removing the FD cache. This version also lays the foundation for new and exciting features coming in HAProxy 2.2. With that said, there are still some amazing features that made it into this release, including Dynamic SSL Certificate Updates, FastCGI, and Native Protocol Tracing!
We’ve put together a complete HAProxy 2.1 configuration, which allows you to follow along and get started with the latest features. You will find the latest Docker images here. This release is, of course, made possible by the work of our open-source community members. They provide code submissions for new functionality and bug fixes, do quality assurance testing, host continuous integration environments, report bugs, and work on documentation. If you’d like to join this amazing community, find it on Slack, Discourse, and the HAProxy mailing list.
- Dynamic SSL Certificate Updates
- Native Protocol Tracing
- Removal of the File Descriptor Cache
- Scheduler Improvements
- Defaulted HTTP Representation to HTX
- Fetches and Converters
- Deprecated Configuration Options
- Strict Limits Setting
- Debugging for Developers
- Version Info Shows Links
- Runtime API Field Descriptions
- Prometheus Improvements
- Miscellaneous Improvements
Dynamic SSL Certificate Updates
SSL certificate information has been centralized so that if the same certificate file is referenced by multiple
bind lines, it is only loaded once. This means fewer system calls and a substantially faster startup time when loading the same certificate on multiple lines. There are some users who load tens of thousands or even hundreds of thousands of certificates multiple times and they will see a visible improvement.
We have also added the ability to update SSL certificates using the Runtime API. Use the
set ssl cert and
commit ssl cert commands. Here’s an example:
A direct outcome of supporting multiplexed connections everywhere was being within striking distance of supporting FastCGI. It became so tempting that we had to give it a try, and we succeeded! The new FastCGI support in 2.1 can help you improve resource management by eliminating superfluous components.
For example, you can remove components such as web servers deployed solely to serve as HTTP-to-FCGI gateways. Such components add a small amount of latency and obstruct HAProxy’s view of the backend application. Without them, HAProxy has a clear view of the application’s response times and can use it to operate at full speed without overloading it.
A new section named
fcgi-app defines parameters for communicating with a FastCGI application. A normal
backend section can then relay requests to it by including the
use-fcgi-app directive and setting
proto to fcgi on the
Native Protocol Tracing
HAProxy has a great track record of providing diagnostic and observability information that can be used during debugging of problematic requests and responses. To make it even better, a new tracing infrastructure has been integrated, allowing systems engineers and developers to collect low-level trace messages from HAProxy without having to recompile it.
Traces can be enabled by using Runtime API commands and will report diagnostic information at varying levels of detail. The messages can be sent to stdio, log servers, or to ring buffers, which are a rotating log. In the following example, we use the
show trace command to see supported trace sources and their current states:
There are a few variants of the
trace command. When given a source and level, it enables debug tracing for the given source (e.g. h2) and filters down to events for the specified level. Use the sink variant to configure where trace messages are sent (e.g. stdout and stderr). Use the start/stop/pause variant to start, stop and pause tracing.
The following options are available for the level. It is recommended to only use “user” unless you have a reason otherwise.
|user||This will report information that is suitable for use by a regular HAProxy user who wants to observe their traffic.|
|proto||In addition to what is reported at the “user” level, this also displays protocol-level updates. This can, for example, be the frame types or HTTP headers after decoding.|
|state||In addition to what is reported at the “proto” level, this will also display state transitions (or failed transitions) that happen in parsers, so this will show attempts to perform an operation while the “proto” level only shows the final operation.|
|data||In addition to what is reported at the “state” level, it will also include data transfers between the various layers.|
|developer||It reports everything available, which can include advanced information such as “breaking out of this loop” that are only relevant to a developer trying to understand a bug that only happens once in awhile in the field. Function names are only reported at this level.|
The following example starts tracing for HTTP/2 messages at the user level and sends them to stderr:
It can also be handy to see events from all sources at once. This release introduces an in-memory ring buffer (1MB in size) that lets you see messages regardless of their configured sinks. Simply issuing a
show events buf0 from the Runtime API will show everything and adding the –w –N flags after it will work like
In the following example,
show events is called from an interactive shell:
To change the level, you would use:
Removal of the File Descriptor Cache
The file descriptor (FD) cache has been completely removed. This was a leftover from the very old “speculative I/O” mode intended to save a lot of syscalls in callbacks by keeping track of the last polling status. With recent improvements, it’s not needed anymore. This change has shown an increase in performance of up to ~20% on some artificially tailored workloads. However, you can realistically expect to see a 5-10% improvement in a real production environment.
A part of the internal scheduler was improved to support waking up tasks belonging to another thread. This was a necessary step in order to deal with idle inter-thread connections, but also allowed us to simplify and speed up other parts of the code. The scheduler now uses a combination of a locked and a lock-free list to regain 5-10% performance on workloads involving high connection rates.
Defaulted HTTP Representation to HTX
HAProxy 2.1 officially removes support for the legacy HTTP mode and now supports the Native HTTP Representation only, also known as HTX. HTX was introduced in version 1.9 and became the default in version 2.0. No configuration change is required, unless you try to specify
no option http-use-htx in which case you’ll get an error.
[WARNING] 326/104833 (24011) : parsing [haproxy.cfg:9]: option ‘http-use-htx’ is deprecated and ignored. The HTX mode is now the only supported mode.
Several changes and bug fixes have come to HTX since the release of 2.0. Notably, it was reported by users that some legacy applications required case-sensitive header names. There is no standard case for header names, because as stated in RFC7230, they are case insensitive. To alleviate this challenge, and help these legacy applications seamlessly transition to a modern architecture, the new
h1-case-adjust directive was introduced. When used, HAProxy will replace the given header name with the value on the right:
If you have many headers that require adjusting, use
h1-case-adjust-file to load them directly from a file. Each line in the file should contain a lower-case header name and its new value.
Because these directives are set in the
global section, they are visible to all proxy sections. However, you must enable their use for specific proxies and server pools by adding an
option h1-case-adjust-bogus-client directive to a
listen section and/or an
option h1-case-adjust-bogus-server to a
backend. These should only be used as a temporary workaround for the time it takes to fix the client or server. Clients and servers that need such workarounds might be vulnerable to content smuggling attacks and must absolutely be fixed. They may also break randomly if you add other layers or proxies that don’t support uppercase names. Note that this feature has been backported to version 2.0.10.
Another update relates to the implementation of HTTP/2. In the legacy HTTP mode, HTTP/2 was implemented on top of HTTP/1. HTTP/2 requests were turned into HTTP/1 requests in “origin form”:
GET /path/to/file + Host header
But HTTP/2 clients are encouraged to use absolute form:
Our conversion always used the origin form, which resulted in the loss of the HTTP or HTTPS scheme on end-to-end transfers. Now that HTX is the only internal representation, it was possible to keep the request in its original form (typically absolute for HTTP/2 and origin for HTTP/1) and preserve all elements end-to-end.
One visible effect is that logs will now show the scheme, such as “GET https://authority/path” instead of “GET /path”, since the URI really is this. Some will find this better, others may be annoyed, but it’s still possible to change the format if desired. What matters is that we do not lose the scheme anymore. This will also make it easier for users to build redirects that include the scheme.
Fetches and Converters
New fetch methods have been added, as described in the following table:
|srv_name||Returns a string containing the name of the server that processed the request. It can be useful to return this to the client for debugging purposes.|
|srv_queue||Takes an input, either a server name or <backend>/<server> format and returns the number of queued sessions on that server.|
|fc_pp_authority||Returns the PP2_TYPE_AUTHORITY Type-Length-Value (TLV) sent by the client in the PROXY protocol header. See https://www.haproxy.org/download/2.1/doc/proxy-protocol.txt|
|uuid||Returns a universally unique identifier (UUID) following the RFC4122 standard. Currently, only version 4 is supported.|
A new converter,
sha2(number_of_bits), is now available for generating a checksum for a binary string using the SHA-2 cryptographic hash function. The result is a binary value with a byte length equal to number_of_bits / 8. The number_of_bits parameter can be set to 224, 256, 384, or 512. The default is 256.
The SHA-2 function is more secure and reliable than MD5 or SHA-1 because it is more resistant to hash collision attacks. You might use it to sign a cookie and verify the signature later by concatenating the value with a secret key before hashing.
Deprecated Configuration Options
HAProxy attempts to be backwards compatible across versions. However, there comes a time when certain directives are due to be phased out as they are no longer relevant or better methods have been introduced. One of these changes is the removal of
reqrep, which have been emitting deprecation warnings in previous versions for a long time.
These are very old directives that operate upon a whole HTTP line to modify HTTP headers or the URI. They require emulating a dummy request when dealing with HTTP/2 and HTX, which isn’t ideal. It also presented the challenge of not being able to use case-sensitivity on values without having the name. All req* and rsp* directives should be replaced with
http-request replace-uri, and
Also, several deprecated options and directives have been officially removed and will present a fatal error when used. See the table below for the newer directives to use instead.
|option independant-streams||option independent-streams|
Strict Limits Setting
HAProxy communicates with the operating system to increase various system limits as needed, such as the file descriptor limit to account for
maxconn settings. In some cases, HAProxy may not be able to increase these limits and so displays a warning. However, there are times when you may want to force HAProxy to abort at startup if it can not get the required limits.
As an example, you may require high limits in production and if there is an error achieving this your production traffic could be affected. To alleviate this problem, you can add a new option
strict-limits to the
global section. If HAProxy fails to increase the limits set by
setrlimit()—typically the file descriptor limit—then HAProxy will fail to start. This setting will be turned on by default in version 2.3.
Debugging for Developers
debug dev Runtime API command that was previously only available when building with -DDEBUG_DEV is now always available, but only shown when expert-mode is on. This is sometimes needed by developers to extract information about a sick session or to perform fault injection. Do not try to use them in production without being invited to do so, you’ll very likely crash your process before you understand what you did.
Version Info Shows Links
Users can now get a better view of the status of the code they’re using by running
haproxy -v, as shown here:
This gives you a link to a page listing the issues relevant for the version of HAProxy you are running. This makes it easy to see whether or not you are affected by a bug.
Runtime API Field Descriptions
show info and
show stat Runtime API commands accept a new parameter,
desc, that adds a short description to each field.
In certain environments, exporting all of the metrics available by Prometheus can result in a large amount of unnecessary data. A method was developed to allow for exporting only specific sets of data. In HAProxy 2.1 you can now pass the
scope parameter within the query string to filter exported metrics. It accepts the following values:
- * (all)
scope parameter with no value will filter out all scopes and nothing will be returned. If multiple
scope parameters are set, they are parsed in the order they appear in the query string. Here are a few examples:
|/metrics?scope=server||server metrics will be exported|
|/metrics?scope=frontend&scope=backend||Frontend and backend metrics will be exported|
|/metrics?scope=*&scope=||no metrics will be exported|
|/metrics?scope=&scope=global||global metrics will be exported|
Several miscellaneous improvements have been made:
- The storage of the server-state global file was moved to a tree, which provides much faster reloads.
http-response sc-set-gpt0now accept an expression.
resolve-optsnow accepts ignore-weight so that when generating servers with DNS SRV records, server weights can be set dynamically with agent health checks or the Runtime API and not be subsequently reset by DNS SRV weights.
- The Stats page can now be exported in JSON format. This can be accessed by appending “/;json” to the URI.
send-proxy-v2directive can now send the PP2_TYPE_AUTHORITY value, allowing it to chain layers using SNI.
- Added support for the user and group directives to the program Process Manager section.
- Connections should now take significantly less memory as the source and destination addresses are now dynamically allocated as needed. That translates to 128 to 256 bytes saved per connection and per side in the common case.
With all of the architecture improvements, refactoring, and removal of obsolete code, you can expect better performance in some areas, improved debugging capabilities, and even more responsive developers thanks to their ability to work on a cleaner and more manageable codebase. Keep a watch on our blog for other news as we turn our attention back to adding new and amazing features!