Intent-driven, Fully Automated Deployment of Anycasted Load Balancers with HAProxy and Python

Transcript

Intent-driven, Fully Automated Deployment of Anycasted Load Balancers with HAProxy and Python, a long title, but excuse us, we are from University, we have to do something like that. So, who are we?

Johannes Kampmeyer, I’m Sebastian Langenhorst, great introduction by the way and we are from a mid-sized university in the middle of Germany called Paderborn. The figures: We have twenty-something-thousand students, 2,500 employees, different departments, every department stands for itself; If anything has to do with IT, they end up eventually at us, so managed services, hosting, application development, network, email, everything is done by the IMT, the central computing department.

To maintain all this we have quite some infrastructure, we need a load balancer, of course, to do so. We had this quite big, pricey IP solution before that went out of everything, so we had to replace it and we took a look at the market and one product stood out, HAProxy. We try to use HAProxy for everything we do; We do LDAP, we do mail, we do web server, like everything we have, with HAProxy.

What’s our approach? We went and got us two off-the-shelf rack servers from a standing contract; We put quite a lot of resources on them, so it’s a 56 cores, 100 Gigs of RAM because we didn’t know what we would really need. The NICs are 10 gigabit. That’s okay for us since our backbone just has 40, so with two of them we could exploit it. What we do right now, quite an approach that it’s even documented with “Please don’t do it.” We do it anyway. For each VIP, we have a single process with a single config; This allows us to reload, to replace a single VIP without affecting any of the others. We didn’t know if we would really need it. Right now we can do it, we have enough hardware to just do it, some time before it gets a problem.

We went for an active-active setup. With the VIPs, now it’s just around 150, get anycasted to both load balancers, so it’s an anycast structure above HAProxy. How do we do it? We have an anycast health checker that checks if the HAProxy is up and running, then announces it via BIRD to route reflectors and then they will announce it to the network switches.

Even if we do quite something for the university, we are quite a small department, so we rely on heavy automatization. If you can’t automate, automatizate, anything, we can’t write it by hand, that’s just impossible. So, we deploy it via config deployment and what happened to be really nice is the graceful reloading with HAProxy so it doesn’t break anything.

What we do—or with our former solution—everything was routed through the load balancer. We wanted to go back from that so that the routers are routing and the load balancers are load balancing, so we don’t have a single point of failure; We still have a single point of failure, but there’s a…we had a problem that our ACLs on the routers didn’t match on the load balancer. With HAProxy, we wanted to get a solution for this. We use termination on the HAProxy, we make heavy use of the Proxy protocol whenever it’s possible; If it’s not possible and it’s HTTP, we use X-Forwarded-For connections.

Another request or demand—yeah it’s more of a demand—from our web team was: “We don’t want to have anything to do with SSL, with TLS. Please terminate.” So we terminate everything that has something to do with TLS on the HAProxy. That is wonderful for us! For the first time we have a consistent SSL cipher out there, we have a consistent TLS version that we are supporting or not supporting. It’s security, it’s great.

Still, active-active is nice, but we have some interesting products, I will call them, which still requires that we are routing on the HAProxies; Thanks to the TPROXY extension it’s still possible to use HAProxy as a frontend and then transparent the tunnel through HAProxy and route. There’s one product we have that uses client IP as the session key, it’s a real good idea, but we can’t replace it, it’s another department, so we have to work around it.

Okay, in a picture how does it look? In the introduction you had the main application, so with our old load balancer we had the load balancing with the network, our networking people didn’t like it, so with HAProxy we got, and the big application team, as a load balancer. This way we have, it’s nice that we don’t have to do anything with the switches, or with the routers. Sorry, I’m no networking guy.

So, we don’t have to have any access to the routers; we go through the route reflectors; The route reflectors, HAProxy, and the service backends, everything feeds out of one config deployment and whenever something new appears or some change appears, anycast health checker will notice on the HAProxy machines, will then via BIRD tell the route reflectors and those then announce a BGP route to the datacenter routers. So, everything we need and with the route reflectors we have an additional layer of security there.

Okay. Our configuration management is—I don’t know if anybody has ever heard of it—it was unusual, it’s called Bcfg2. It has XML, no concept of services, but of hosts, so it can configure hosts, but no services. It uses groups to manage, actually, the hosts, so we have different groups like webserver and a group type of three, and productive or unstable or testing; and with a layer of these groups, with a subgroup of this group, we can address one machine, so that’s what we have.

It’s implemented in Python so it’s easily extensible; It supports common templating languages, which we make some use of, some heavy use of, and since it’s Python it’s easily extensible to outside use. If there’s some data we can source from anywhere, we can get it within our config management and make use of it. That made us, to get the idea, that it could possibly make some wholesome, automated approach to get HAProxy to run and that’s Johannes’ part.

Thank you, so now my part, the buzzword part. We have an intent-driven architecture, so what would you even call “intent” in that case? For us, our most common use case is for our web development team. They have some kind of requirement and they say, “Okay, please, I want this FQDN balanced to these backends with these options,” and the only way for us to manage this, because we had no way to define this in our current config management, we said okay, we create a consistent definition of a service and all devices and applications that need to be configured simply source this data and we generate the configuration for them. HAProxy is currently deployed by this. We are using it for netfilters, which is nice because when we define, okay this service needs these access ports and this protocol, we simply know okay we don’t need anything else, so close every other connection.

We are heavily monitoring with Icinga2. So if we know a service is running on HTTP, we can check the liveness with active checks on the HAProxy frontend, and also the backends if they are living, and also this works for our mail service and whatever you like because it’s extensible. The best thing is it’s a single source of information, so we simply say if it’s not there it doesn’t exist for us.

How do you call the system if you’re not shipping this as a product? So, it’s the “service magic” and our design approach was pretty simple; We said, okay, we need a filesystem-based approach because we have an implicit hierarchy, and we can give access to different groups on different levels of the tree so they can define their defaults there, and such a defaults file is basically overriding all the defaults set up to this level.

If the web team has another service, you create a new folder, add a new defaults file, and they have defaults for every type of service they want to deploy as such; and the best thing about this is you don’t have to specify all fields that we support because most things we can simply infer. If you have a service file like here on the right, and you see that we have SSL turned on and we say the SSL redirect is false, we know, okay, the HTTP server is doing the redirecting.

Also, the nice thing about this is we have a lot of service owners that want to be in full control. With our previous solution, we had to a web interface and it was tricky, and now we can give them access to the service directory and they say, “Okay, I deploy a new service, it’s called like this, it has this FQDN, and the web server has to listen on this port, and it’s Apache 2, by the way,” and we can simply ship that. Another great thing is we can easily sanity check what they are defining there because if they say this FQDN has to exist there, we can simply check in DNS, and this goes for the groups if they exist or not and, yeah, it helps us with consistency.

This brings us to our HAProxy configuration. As I already said, we are making heavy use of the defaults. So let’s say we have a backend that is in the group typo3cluster-uni-production, we can simply resolve all hosts that are in this group and know, okay, those are our backend hosts, we know the web server is listening on a certain port, and we know HAProxy can reach the web service there, and also netfilter will allow HAProxy to access the web server on that port. We are mostly using templates to generate all these application configurations, which is triggered by the admin or even the service owner. They can say, “Build me this service.” They see, okay, if the following files have changed, maybe you want to review them, or not, and then write them to disk. So, we have a stable configuration, then if the admin is fine with it, they can simply push it out or wait for our hourly deployment.

To the lessons learned about building something like this: A lot can go wrong, so first of all communication is always key, and describing the service is actually pretty hard because sometimes you’re not using the same vocabulary and so we all had to get on the same page, which means which option actually triggers what. We had a case of faulty firmware, which led to an open BGP session that, even though the link was nearly down, anycast would still try to serve packets to one of our HAProxy servers. That went away once we got better firmware, but we will add BFD to improve recovery time and detection.

Also, automation always has a price in business complexity. Getting someone up to the level to understand how we are doing things is, yeah, it takes some time and it’s not like, “Yeah, here you have your Ansible playbook; Deploy this and you’re done,” but, and closing, things are not as simple as you might think.

But what actually are the benefits? This is a wide variety; We have currently a consistent deployment for everything and this is the first time ever we know, okay, we have machines where other departments can get root, but we know they are consistently defined and this even goes to the monitoring because we…in the past we had systems that weren’t even monitored and then we get a call, “Hey, why is my machine down?” and we said, “Oh, we didn’t even know this existed.”

The IPv6 migration is really easy because we can simply turn on, give the FQDN an IPv6, build the configuration once again so HAProxy listens on that IP, it gets automatically announced by BIRD, and then we’re done because HAProxy is an IPv6/4 gateway, we don’t even need IPv6 for the backend. We have fast reconfiguration because if I change something in this defaults file, I can rebuild the whole batch of servers in seconds, and it’s really cheap. It’s really a cheap solution if we compare it to our old BIG-IP solution; and also we have no vendor lock-in and what I like best is it’s future-proof. If we ever say, “Okay, we are moving to the cloud or we are doing Kubernetes or some other fancy technology, we have a consistent definition of what a service is and we can simply apply this to our new technology because we have written it once, so we can simply reuse it. Also, security is always a nice benefit!