Learn how to use the HAProxy Kubernetes Ingress Controller when hosting multiple tenants in your cluster and how to configure namespaces, access controls, and resource quotas.
Learn more by registering for our webinar: “HAProxy Skills Lab: Building a Multi-tenant Kubernetes Cluster“
Want to supercharge your ingress routing with HAProxy? Download our free eBook now!
This is the third post in a series about HAProxy’s role in building a modern systems architecture that relies on cloud-native technology such-as Docker containers and Kubernetes. Containers have revolutionized how software is deployed, allowing the microservice pattern to flourish and enabling self-healing, autoscaling applications. HAProxy is an intelligent load balancer that adds high performance, observability, security, and many other features to the mix.
It’s a rare bird, a Kubernetes cluster that serves only a single tenant. In the wild, you’ll likely encounter clusters where tenants are packed in close: QA and Dev, Team A and Team B, Java application and .NET application—environments, teams, and technology stacks declare their stakes on resources. It’s essential that you plan ahead for multiple tenants, set up the proper namespaces, define access controls, set resource quotas, and configure ingress routing.
Sharing resources in a Kubernetes cluster is a logical way to save money on the cost of infrastructure. In this post, we’ll share tips for setting up multiple tenants and, in particular, how to configure the HAProxy Kubernetes Ingress Controller to serve traffic to multiple tenants.
Namespaces are Key
A Kubernetes namespace groups objects inside of a shared scope, providing a sandbox where objects created by one tenant don’t overlap with objects created by another. Take for example a Dev and a QA environment. You can host both environments inside of a single Kubernetes cluster where they can share server resources, yet remain oblivious to one another. Each environment, or “tenant”, can duplicate your entire application stack. Building walls around each tenant avoids accidentally exposing an experimental Dev service within the QA environment, deleting the wrong object, or applying a breaking change to the wrong application
Declare a new namespace by adding a YAML file that defines a Namespace object, like this:
In this instance, the namespace is named dev. Use
kubectl to apply this change to your cluster:
Once created, add objects to the namespace by referencing its name within the object’s metadata. In the following example, a ConfigMap object is added to the dev namespace:
Only objects within the same dev namespace will have access to this ConfigMap. Also, when using the
kubectl command-line utility to view objects, you will need to include the
--namespace argument or else the returned list will come up empty:
kubectl get namespaces command to view all of your defined namespaces:
Managing User Access to a Namespace
Once you’ve defined a namespace, you can configure role-based access control (RBAC) to limit who has access to it. Out of the box, there are already a few roles defined, including admin, edit, and view. In the following sections, a new user login is created and given the edit role in the dev namespace, which gives it read/write access to that namespace only.
Add a RoleBinding
First, create a new RoleBinding object that assigns the edit role to a user named bob.
The edit ClusterRole is already defined and can be scoped to the dev namespace by setting the namespace metadata field. Who is Bob? It’s a user who isn’t represented as an object per se (there is no User object in Kubernetes), but who will authenticate to the cluster using a client certificate that contains a CN field set to bob. You can also grant permissions to a group of users. In the following RoleBinding object, a group named dev-group is granted edit access to the dev namespace:
For group permissions, when you create the client certificate, its O field must match the subject name, which is dev-group in this case. Use
kubectl to create the object in Kubernetes:
Create a Client Certificate
The next step is to create a certificate signing request (CSR) for a new client certificate. There are a number of tools that you can use to do this, such as the
openssl command-line utility. In the following example, I use
openssl to create a CSR for a user named bob with a group of dev-group to demonstrate setting the CN and O fields:
Next, you’ll need to sign the CSR with your cluster’s CA certificate in order to get a client certificate. I’m using Minikube in my test lab, so I could sign the certificate signing request with Minikube’s CA certificate and key, which can be found in the ~/.minikube directory. I would use the following
openssl command to create a client certificate named bob.crt.
Another way to sign the CSR and get a bob.crt file is to use the Kubernetes Certificates API, wherein you create a CertificateSigningRequest object. You will need to store the CSR data in a YAML file as a base64-encoded string and then apply the YAML file to your cluster, so it’s easiest to do it from the command line, like this:
Then, approve the CSR:
You can then download the signed certificate with the
kubectl get csr command:
Add a Cluster Context
Next, add a new cluster context that lets you log in as bob, using the bob certificate.
You’re now using the minikube-bob context to access your Minikube Kubernetes cluster. If you try accessing or creating objects in the dev namespace, it will work, but you’ll get an error if you try to access an object in any other namespace.
You can switch back to the normal Minikube context, which has admin privileges, like this:
An Ingress Controller that Watches a Namespace
Now that you’ve created a namespace and given limited access to it, let’s see how to manage HTTP traffic going into the environment. The HAProxy Kubernetes Ingress Controller brings the power of HAProxy to Kubernetes, allowing you to leverage its high performance, reliability, and security.
Be sure to switch back to the normal admin context before going further. Without any special configuration, our Kubernetes Ingress Controller will watch over all namespaces. When a pod is added or removed anywhere within the cluster, the controller is notified, which means that any of your teams can use it for ingress traffic routing. That’s great news if you want to set up routing quickly for all of your teams (i.e. tenants). However, there are a few reasons why you may decide to deploy multiple ingress controllers.
For one thing, by creating multiple ingress controllers, you can apply a walled garden approach for each tenant. By creating a distinct ingress controller for each one, you can:
- collect distinct HAProxy metrics per tenant, such as request rates and error rates.
- set rate limits per tenant to prevent “noisy neighbors” syndrome.
- define custom timeouts per tenant to accommodate varying SLAs.
- reuse the same URL paths to keep your applications consistent between tenants.
When you deploy an HAProxy Kubernetes Ingress Controller using Helm, add
--namespace-whitelist to the controller.extraArgs field to set the namespace to watch, as shown:
You can specify more than one namespace to watch:
This Ingress object is created within the dev namespace and, therefore, is picked up automatically by an ingress controller that has its whitelist set to dev:
You could create an identical Ingress object in your qa namespace, but it won’t route through this particular ingress controller because its namespace is different. Each ingress controller can be exposed to a unique IP address so that tenants can be given their own subdomain. It won’t be possible for one tenant’s traffic to mix with that of another. Note that you can use Network Policy objects to restrict access between services inside the cluster.
An Ingress Controller that You Target
Another way to manage ingress routing is to use ingress classes. Whereas
--namespace-whitelist tells the ingress controller to watch a specific namespace for changes, an ingress class flips that responsibility around, giving an Ingress object a chance to target the controller it wants by name. To set this up, add class to the list of arguments when defining your ingress controller. Here, the
--ingress.class argument is set to intranet:
Maybe this ingress controller exposes services only to the company’s intranet. You may have another ingress controller that has a class of public, exposing services to external customers, for example. An Ingress object targets its desired controller by setting its haproxy.org/ingress.class annotation, as shown:
This puts the control into the hands of your service developers. They can choose which ingress controller to use and it gives them a greater degree of autonomy. You can even use this with multiple tenants if you don’t mind giving them a common IP address for accessing their services.
As a final tip, each namespace can be assigned its own allotment of resources. For example, QA might be allocated more or less CPU and memory than Dev. This lets you prioritize which tenants receive the resources, or lets you simply keep things equal for everybody. If you don’t do this, then you risk one tenant utilizing more than their fair share and leaving other tenants squabbling over the scraps.
It is essential then that every pod defines how much CPU and memory it needs so that Kubernetes knows when a tenant is about to exceed its resource limits. We won’t go into detail about this, but this is accomplished by setting requests and limits on a pod, which you can learn more about on the Managing Compute Resources page. You can also create defaults for a namespace, in case a pod doesn’t set its own limits, by creating a LimitRange object.
Let’s cover how to set the resource quota for a namespace, which determines the cap on resources. If a tenant requests more resources than what you’ve allowed here, their objects won’t be created. Resource quotas let you restrict:
- the total CPU that can be used
- the total memory that can be used
- the amount of hard drive storage that can be used
- the number of objects that can be created
The following ResourceQuota object sets limits for CPU and memory in the dev namespace:
kubectl to apply the quota:
Then, you can view how much has been used so far:
Quotas will help keep tenants from overusing resources and allow you to see how much a particular tenant has used so far, which is great when planning whether you need to expand the cluster. It’s a vital step when planning for multiple tenants.
In this blog post, you learned some tips for managing multiple tenants that share resources within a Kubernetes cluster. The HAProxy Kubernetes Ingress Controller lets you whitelist certain namespaces to watch so that each namespace can be routed through a specific controller. You can also target specific ingress controllers by using ingress classes.
When setting up multiple tenants, it pays to configure RBAC and to give teams access to only their respective namespaces, which you can accomplish by using client certificates. You should also consider setting resource quotas to prevent a tenant from using more than their fair share of CPU and memory.
The Enterprise version of the ingress controller combines HAProxy, the world’s fastest and most widely used open-source software load balancer, with enterprise-class features, services, and premium support. Contact us to learn more and sign up for a free trial.