HashiCorp added a service mesh feature to Consul, its service-discovery and distributed storage tool. In this post, you’ll see how HAProxy is the perfect fit as a data plane for this architecture.
HAProxy is no stranger to the service mesh scene. Its high performance, low resource usage, and flexible design allow it to be embedded within various types of service-oriented architectures. For example, when Airbnb needed to scale its infrastructure to support a growing number of distributed services, it developed SmartStack. SmartStack is a service mesh solution that relies on instances of HAProxy relaying communication between services. Using HAProxy allowed SmartStack to take advantage of advanced load-balancing algorithms, traffic queuing, connection retries, and built-in health checking.
HAProxy Technologies is working with HashiCorp to bring you a Consul service mesh that utilizes HAProxy as a data plane. This will allow you to deploy the world’s fastest and most widely used software load balancer as a sidecar proxy, enabling secure and reliable communication between all of your services.
In Consul 1.2, HashiCorp released Connect, which is a feature that allows you to turn an existing Consul cluster into a service mesh. If you’re familiar with Consul, you’ll know it for its distributed key/value storage and dynamic service discovery. With the addition of Connect, you can register sidecar proxies that are colocated with each of your services and relay traffic between them, creating a service mesh architecture. Best of all, it’s a pluggable framework that allows you to choose the underlying proxy layer to pair with it.
With Connect, HAProxy can be combined with the Consul service mesh too. This provides you with many advantages when designing your architecture. First, let’s take a step back and cover what a service mesh is and why it benefits you to use one.
Why Use a Service Mesh?
Why would you use a service mesh anyway? It comes down to the challenges of operating distributed systems. When your backend services are distributed across servers, departments, and data centers, executing the steps of business processes often involves plenty of network communication. However, if you’ve ever had the responsibility of managing such a network, L. Peter Deutsch’s Fallacies of Distributed Computing will ring true. The fallacies are:
The network is reliable.
Latency is zero.
Bandwidth is infinite.
The network is secure.
Topology doesn’t change.
There is one administrator.
Transport cost is zero.
The network is homogeneous.
The network is not always reliable. Topology does change. The network is not secure by default. How can you manage these risks? Ultimately, you want to apply mechanisms that address these issues without adding tons of complexity to every service that you own. In essence, each application shouldn’t need to be aware that a network exists at all.
That is why service meshes are becoming so popular. They abstract away the network from the application code. A service mesh is a design pattern where, instead of having services communicate directly to one another or directly to a message bus, requests are first passed to intermediary proxies that relay messages from one service to another.
The benefit is that the proxies can support functions for dealing with an unruly network that all services need but which are more easily handled outside of the services themselves. For example, a proxy could retry a connection if it fails the first time and secures communication with TLS encryption. You end up separating network functionality from application functionality.
Capabilities within HAProxy mitigate the network-related problems that we mentioned before. For example, the solutions include:
The network is not reliable = Retry logic for connecting to services
Bandwidth is not infinite = Rate limiting to prioritize certain traffic or prevent overuse
Topology changes = Consistent endpoints, such as always communicating with localhost, service discovery, and DNS resolution at run time
The network is not secure = Truly end-to-end encryption with mutual TLS authentication
There is never just one administrator = Authorizing which services are allowed to connect to which other services
Transport cost is not zero = Observability of service traffic
Let’s see how the pieces fit together.
Control Plane & Data Plane
How does Consul relate to HAProxy? What are the responsibilities of each? Think of the whole thing like a real-time strategy video game. One of my favorites is Starcraft. In this game, you, the player, control an army of workers whose sole mission is to await marching orders from you and then execute them. For example, you could tell them to go mine for minerals or explore a new area of the map, and they’ll happily go off and do what’s been asked of them.
Compared to a service mesh, you, the player, represent Consul. Consul gives the marching orders to all of the proxies under its influence. It provides each one with the configuration details it needs: information about where other services can be found, which port to listen on for incoming requests, which TLS certificate to use for encryption, and which upstream services the local service depends on. In service mesh terminology, Consul is the control plane.
HAProxy, the advanced proxy
The proxies take on a more drone-like approach to life. You don’t need to configure each instance of HAProxy individually. They are responsible for querying the central Consul registry to get their configuration information on their own. This proxy layer is called the data plane. Of course, they are just as important as your video game workers. The proxy technology you choose determines your service mesh's capabilities. Will it be able to encrypt communication? Will it have logic for reconnecting to a service if the first attempt fails?
HAProxy gives you features that a distributed architecture requires. It is the product of the hard-working open-source community and has become known as the fastest and most widely used software load balancer in the world. You get TLS encryption, connection retry logic and authorization between services, all out-of-the-box and in a lightweight process.
The Components
The key pieces of your service mesh will include the following:
Your service
A Consul agent, running locally that your service registers with
The proxy, which is registered as a sidecar
A quorum of Consul agents that are in server mode, responsible for managing the service registry and configuration information
Let’s cover these components in more detail.
Your service (aka your business logic)
Your service is at the heart of the design. It exposes functionality over a TCP port and, optionally, relies on other distributed services. The goal is to minimize the ways in which you need to change it to fit into the service mesh. Ideally, it should continue as it always has, oblivious to the topology of the outside network. Consul gives you this separation of concerns.
Like in our video game analogy, let’s say we’re talking about a mining service for minerals. However, it may need to talk to the map service to find where to start working. Ordinarily, you would need to configure it with the map service's known address or DNS name. If these settings ever changed, your mining service’s configuration also had to change to point to the new endpoint.
With Consul, service discovery allows you to query a central registry to discover where services live. However, with Connect, it gets even better. A local instance of HAProxy is created next to the service to listen on localhost. Then, your service always queries it as if it were the remote map service. This is known as a sidecar proxy: Each service gets its local proxy and, through the relaying of messages, can reach other remote services as if they too, were local. Now, you can point your local service’s configuration at localhost and never need to change that endpoint again.
Consul agent
Also local to your service is a Consul agent. An agent is the Consul executable running in regular agent mode instead of server mode. Register the local service with this agent by adding a JSON file to the /etc/consul.d folder. This gets synced to the server-mode agents.
Think of it as a walkie-talkie to the agents running in server mode, which holds the source of truth. The registration gets sent up and saved to the global registry. Then, when HAProxy wants to discover where it can find services on the network, it asks its local Consul agent, and that agent pulls the information back down. Afterward, it gives the answer to your proxy.
.png)
A locally run Consul agent is one of the key components of the service mesh
The same goes for the upstream services you’re calling. They each get their own local Consul agent. They register with it so that others can find them and tell Consul about any services they, in turn, depend on. All of this information is used to configure the proxies so that, in the end, all services talk to localhost, and all communication becomes proxy-to-proxy.
The HAProxy Sidecar
Next to each service and Consul agent is a sidecar proxy, an instance of HAProxy. You don’t configure it directly. Instead, you install it with a specialized handler that queries the Consul agent to know which upstream services your local service depends on. Then, HAProxy sets up listeners on localhost so that your application can talk to the remote endpoints that it needs to, without knowing exactly where those endpoints live. In essence, you’re abstracting away the network.
A benefit to routing traffic through a local proxy is that the proxy can enforce fine-grained authorization rules. Consul lets you define intentions, which are rules that govern whether one service can talk to another. HAProxy queries the local Consul agent at runtime to check if an incoming connection is allowed.
To review, your service registers with its local Consul agent information about itself and the upstream services on which it depends. That agent sends that information to the Consul servers, which maintain the central registry. The local instance of HAProxy then asks the local Consul agent for configuration information, and the agent pulls back the data that the Consul servers have compiled.
HAProxy then configures itself, listening for incoming requests to the local service and also for outgoing requests that the service makes to other services. The HAProxy handler continues to check for changes to service registrations and updates itself when needed.
.png)
Each service and Consul agent have a sidecar proxy, an instance of HAProxy
In the end, all service-to-service communication ends up going through proxies. You can see why choosing a proxy implementation that meets your needs is important. When it comes to speed and the right mix of features, HAProxy has a lot of benefits.
Server-mode agents
You’ve probably got a good idea about how the Consul agents running in server mode fit into this architecture. They maintain the service registry, which is the source of truth about where each service endpoint can be found. They also store each proxy's settings to self-configure itself, such as TLS certificates. The Consul servers host the Consul HTTP API that local agents use to send and receive information about the cluster.
Agents in server mode must elect a leader using a consensus protocol that is based on the Raft algorithm. Therefore, you should dedicate an odd number of nodes, such as three, to participate in the quorum. These agents should reside on separate machines, if possible so that your cluster has resiliency.
The Implementation
Baptiste Assmann presented a proof of concept solution for integrating HAProxy with Consul at the 2018 HashiConf conference in San Francisco.
During his presentation, he demonstrated using HAProxy as a sidecar configured using information from the local Consul agent. Read more about the finalized solution in our blog post Power Your Consul Service Mesh with HAProxy.
Conclusion
Consul’s Connect feature enables you to transform a Consul cluster into a service mesh. Connect is a pluggable framework that allows you to choose the best proxy technology that fits your needs.
In this blog post, you learned how HAProxy can be selected as the data plane, giving you access to features like TLS encryption, connection retry logic, and authorization. I hope you’re as excited as we are about the possibilities that this creates!
Want to stay in the loop about blog posts like this? Subscribe to this blog or follow us on Twitter. Want to see what HAProxy Enterprise has to offer? Contact us to learn more, or sign up for a free trial today! Our expert support engineers have helped some of the largest companies in the world implement service-oriented architectures.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
