HAProxy Technologies’ R&D has released a patchset that allows DNS to be utilized for service discovery in HAProxy. The patchset has already been merged into the HAProxy 1.8 development branch and will soon be backported to HAProxy Enterprise Edition 1.7r2.
HAProxy is a mature, high-performance software component that’s been reliably serving the load balancing and ADC markets for over 15 years now. Over time, the role and functionality required from HAProxy has evolved significantly. When HAProxy was first released, application architectures were quite static and very few configuration changes occurred during runtime. Nowadays, to satisfy the requirements of modern cloud and microservices architectures, an ADC must be very flexible and it must respond to changes in its environment during runtime. Typically, this includes the requirements to:
- Follow servers as they move from one IP to another
- Enable dynamic scaling of backend servers
This is especially important in environments such as Amazon AWS or Kubernetes, where a scheduler is monitoring the CPU and memory usage on individual backend servers and can decide to expand or reduce the service footprint at any given point in time.
One way to update the HAProxy configuration during runtime is through the HAProxy’s Runtime API. That topic was nicely covered in one of our previous blog posts – Dynamic Scaling for Microservices with the HAProxy Runtime API.
Another way to update the configuration during runtime is by using DNS for service discovery, which will be explained in this post.
Please note that the two solutions are not mutually exclusive: the DNS approach allows changing the servers’ statuses, IP addresses, ports, and weights. The Runtime API allows for even more elaborate configuration updates (agent related configs, health checks, maxconn, ACL and MAP statements, etc.).
An important advantage of using DNS for service discovery is that there are no explicit runtime configuration changes or configuration file updates that need to take place. With the Runtime API, custom software or a script is required to invoke the Runtime API and update the runtime and/or on-disk configuration.
According to Wikipedia: “[Service discovery] is the automatic detection of devices and services offered by these devices on a computer network”.
In essence, a service registry is used to monitor the network and maintain an accurate list of nodes where individual services are available. This allows other components in the network to perform efficient service discovery even when changes in configuration occur very often (such as in cloud or microservices environments).
The methods that client components use for querying service discovery systems for information can vary. Two typical ones are:
- JSON/REST interface (usually restricted to a single service discovery system)
- DNS – the Domain Name System
Why DNS for Service Discovery?
There are a couple advantages to using DNS for service discovery:
- DNS is a well established standard (happy 30th birthday in November 2017!)
- It makes HAProxy compatible out of the box with all service discovery systems that can expose discovery information via DNS
- There is no need to implement support for a custom API to discover services
- It is resilient. Don’t believe the claims that a failure in DNS may lead to a global infrastructure failure – it is up to the client components to deal with that
The disadvantages of using DNS for service discovery are:
- When using A or AAAA DNS records, only the IP address of a node can be updated
- When using SRV DNS records, only the IP address, network port, and server weight of a node can be updated
- There may be a delay between the moment a node failure is detected and the moment the clients execute their DNS service discovery queries (while most JSON/REST implementations use long-polling mechanisms to trigger updates on the clients instantaneously)
DNS support in HAProxy
The following section provides a summary of the DNS related features in HAProxy.
In HAProxy 1.5, we’ve added the ability to resolve server hostnames to IP addresses using the system’s libc resolver (and the resolve.conf file). The resolution was done at startup and it couldn’t be repeated at runtime since the libc’s getaddrinfo() call is a blocking call, and it would break the even-driven nature of HAProxy.
In HAProxy 1.6, we’ve introduced a full implementation of an event-driven DNS client that was able to perform DNS resolutions and update HAProxy during runtime. The resolution that was happening during configuration parsing was still required and was done by the libc’s getaddrinfo() call mentioned above.
Only CNAME, A, and AAAA response types were supported, and the runtime resolutions were triggered by the check task (no health checks meant no runtime DNS resolution). Also, each resolution was autonomous and atomic; there was no caching and multiple DNS resolutions for the same hostname were running in parallel.
In HAProxy 1.7, we’ve enabled HAProxy to bypass the libc’s DNS resolution at configuration parsing time (see the init-addr server’s parameter).
And finally, in HAProxy 1.8 (which is expected to have a stable release in November 2017), we have introduced support for SRV query types, EDNS payload size (DNS responses up to 8K), and a dedicated timer for obsolete records. Since this version, the DNS resolution task is also autonomous and works independently of the servers’ health checks, and a DNS response cache exists (multiple objects needing to resolve the same hostname consume one query and use the same response).
DNS SRV Records
Popular service discovery systems can export data using the DNS SRV query type.
The SRV records are contained in the ANSWER section of the DNS responses and have the following structure:
_service._proto.name. TTL class SRV priority weight port target
Where the description of the fields is as follows:
- _service: standard network service name (taken from /etc/services) or a port number
- _proto: standard protocol name (“tcp” or “udp”)
- name: name of the service — the name used in the query
- TTL: validity period for the response (this field is ignored by HAProxy as HAProxy maintains its own expiry data which is defined in the configuration)
- class: DNS class (“IN”)
- SRV: DNS record type (“SRV”)
- priority: priority of the target host. Lower value = higher preference (this field is ignored by HAProxy, but may become used later for indicating active / backup state)
- weight: relative weight in case of records with the same priority. Higher number = higher preference
- port: port on which the service is configured
- target: canonical hostname of the machine providing the service, ending in a dot
Usually, the DNS server will also return the resolution for the targets mentioned, and it will provide that information in the ADDITIONAL SECTION.
Resolution in Practice
Putting the theory into practice, let’s say we have a service called “red”. It is delivering an HTTP application on port 8080, configured on four multiple servers or containers in our architecture.
To get the list of servers, HAProxy will perform a DNS query equivalent to the following shell command:
dig -t srv _http._tcp.red.domain.local
And the service discovery system will answer with the following records:
;; QUESTION SECTION: ;_http._tcp.red.domain.local. IN SRV ;; ANSWER SECTION: _http._tcp.red.domain.local. 30 IN SRV 10 25 8080 3963643338366463.red.domain.local. _http._tcp.red.domain.local. 30 IN SRV 10 25 8080 3366376637306635.red.domain.local. _http._tcp.red.domain.local. 30 IN SRV 10 25 8080 3464316362303933.red.domain.local. _http._tcp.red.domain.local. 30 IN SRV 10 25 8080 3963326437356461.red.domain.local. ;; ADDITIONAL SECTION: 3963643338366463.red.domain.local. 30 IN A 10.1.29.2 3366376637306635.red.domain.local. 30 IN A 10.1.29.3 3464316362303933.red.domain.local. 30 IN A 10.1.71.2 3963326437356461.red.domain.local. 30 IN A 10.1.71.3
Upon receiving this response, HAProxy will configure itself to the number of records returned, modulo it’s configuration (mainly the DNS related hold timers).
It may happen that a service is configured on 50, 100, or more servers. In such cases, the corresponding DNS responses will be large and exceed the default 512 bytes limit allowed by DNS. HAProxy has support for accepting EDNS payloads (DNS responses of up to 8K in size) and is parsing these answers entirely and correctly.
By using DNS for service discovery in HAProxy, we can conveniently scale backend servers without making explicit runtime configuration changes or configuration file updates.
If you would like to use DNS for service discovery before waiting for the stable release of HAProxy 1.8, please see our HAProxy Enterprise Edition – Trial Version or contact HAProxy Technologies for expert advice on how to best integrate the solution into your existing infrastructure.
In parallel with continuous technical work and improvements, we are also planning on publishing further blog posts describing effective use of HAProxy in microservices environments and integration of HAProxy with 3rd party tools and orchestration systems.
Stay tuned for more blog posts on using microservices with HAProxy, and happy scaling!