Announcing HAProxy Kubernetes Ingress Controller 1.7

We’re proud to announce the release of version 1.7 of the HAProxy Kubernetes Ingress Controller!

In this release, we added support for custom resource definitions that cover most of the configuration settings. Definitions are available for the global, defaults, and backend sections of the configuration. This promotes a cleaner separation of concerns between the different groups of settings and strengthens the validation of those settings. It also makes it easier to deprecate features in the future and allows multiple versions of the custom resources to coexist within the same cluster.

Also, Docker images for multiple architectures, including Arm, are now available, expanding support for running the ingress controller in Kubernetes on various host operating systems. You can also download the ingress controller as a binary from the project’s GitHub Releases page, which is needed for external mode. The ingress controller version 1.7 is built with the latest release of HAProxy, version 2.4.

In this blog post, you will learn more about the changes in this version.

Custom Resource Definitions

Since its inception, the HAProxy Kubernetes Ingress Controller has relied heavily on annotations to tune the behavior of the underlying HAProxy load balancer. It supports annotations on the Service and Ingress resource types for per-application settings, and similar key-value fields in a ConfigMap for configuring global settings.

For those of you not familiar with them, annotations are key-value settings that you can add to the metadata section of a Kubernetes object. For example, to enable HTTP health checking of the URL path /health, you would add the haproxy.org/check and haproxy.org/check-http annotations to the metadata section of your Service definition:

When Kubernetes introduced Custom Resource Definition (CRD), we realized this alternative had a lot to offer our controller. First, CRDs facilitate a cleaner separation of concerns between the different parts of the ingress controller’s configuration; a single CRD can handle all of the global settings, for example. Also, it keeps native Kubernetes resources free of annotations and centralizes configuration settings in one place.

The latter point has many advantages on the technical side. If we exclude the exceptional case of dependent CRDs, a custom resource aggregates settings into one location. That means that when a user submits one of the custom resources, it triggers a single event in the controller’s events loop. In contrast, annotations are scattered across the watched resources and they trigger many events in the code. Custom resources clearly improve the atomicity of how we detect and handle events.

Custom resources also benefit from having better validation. They allows us to specify data types for each setting (e.g. strings, integers, enums, booleans, arrays, and objects) and validate that the submitted value matches. We can also validate other attributes of the CRD’s OpenAPI schema, such as checking whether required fields have been set and perform those checks early on when the user submits the CRD.

Checking that the CRD is coherent as a whole is also made simpler by the all-in-one-place nature of custom resources. We can more easily check fields and their coherency with other, related fields at runtime, in controllers or admission controllers, since the desired state is totally searchable. The illustration below shows the extra validation we can do with a CRD compared to a ConfigMap object.

Third, different versions of a CRD can cohabitate, which is much more difficult to achieve with annotations. Also, deprecating older settings is more manageable. Last but not least, custom resources are first-class citizens at the same level of native Kubernetes resources, with the capacity to be added and removed from the cluster using the same tools.

These observations clearly advocate for adopting CRDs in the HAProxy Kubernetes Ingress Controller. With this release, we offer custom resources for the global, default, and backend sections of HAProxy. Please note that you’re not required to switch to using CRDs. Your ingress controller will still function without them.

Now, let’s see some examples of the new HAProxy custom resources, for illustration purposes.

If you’re interested in learning more, check out Kubernetes Ingress Controller Documentation.

CRD Examples

When you start version 1.7 of the HAProxy Kubernetes Ingress Controller, the logs will display this message:

Custom API core.haproxy.org not available in cluster.

This indicates two things: You have no HAProxy CRD present in your cluster and you’re still using annotations for globals and defaults management. To install the definitions of globals and defaults into your cluster, run the following kubectl apply commands:

This has two effects. The previous warning message disappears and now global and defaults annotations are disabled in favor of custom resources. You can check for the presence of the CRDs with the kubectl get crd command.

Globals

Let’s create a simplified global custom resource. Start by creating a YAML file named myglobal.yaml and copy the following contents into it (change the namespace if needed):

Then submit it with the kubectl apply command:

Next, to ensure that the ingress controller uses only a single object for global settings, update the controller’s ConfigMap object so that it sets the cr-global field to the location of your CRD:

If you were to get a shell prompt into the running ingress controller container, you would see in the resulting HAProxy configuration file something similar to this global section:

The table below lists the fields that you can configure in a global custom resource.

chroot	group	hard_stop_after
nbproc	ssl_default_bind_ciphersuites	log_send_hostname
maxconn	server_state_base	tune_ssl_default_dh_param
nbthread	ssl_default_server_options	ssl_default_server_ciphers
cpu_maps	runtime_apis	stats_timeout
user	ssl_default_bind_ciphers	ssl_default_bind_options
external_check	ssl_default_server_ciphersuites	ssl_mode_async
lua_loads

The mapping to the HAProxy configuration is straightforward, as you can see in the example. You can count in this table 22 fields, with many of them introducing new capabilities that were not available as annotations before. Note that global-config-snippet, which lets you define a raw chunk of configuration markup to be inserted into the HAProxy configuration file, is not supported in the global CRD since all settings are now individually configurable. If you have a global-config-snippet, then translate it into the equivalent fields in your global custom resource.

Defaults

Having more than 50 entries in the defaults CRD permits a complete configuration of the HAProxy defaults section. For this example, create the file mydefaults.yaml with the following contents (change the namespace if needed):

Then apply the configuration to the cluster:

For the exact same reason as with global, you specify the defaults custom resource instance you want to apply with an additional field in your configmap:

This example produces the following defaults section in the HAProxy configuration:

Backends

A backend CRD completes this first batch of newly introduced CRDs. There are simply too many possibilities with this CRD to provide a comprehensive description. We strongly encourage you to browse the definition to get an idea of all these possibilities.

How does it work? You create deployments, services and ingresses exactly as before, but now you can add the cr-backend annotation to services, ingresses or to the global ConfigMap in order to use your backend custom resource. The ingress controller uses the first backend CRD it finds in the following order: ConfigMap, Ingress and then Service.

The backend custom resource lets you customize the backend configuration. In this simple example, we’ll simply change the balance algorithm of the default backend. Create a file named mydefaultbackend.yaml with the following contents (rename the namespace if required) :

As usual, we apply it to the cluster with:

In our cluster, the default backend service is default/kubernetes-ingress-default-backend. Add the following cr-backend annotation to your ConfigMap to attach this CRD to the default backend service:

The backend in the HAProxy configuration then has a modified load balancing algorithm:

Distribution of connections to services/pods

Let’s cover some of the other updates in the release.

The pod-maxconn annotation, which limits the maximum number of connections that the ingress controller will send to a pod, now factors in the number of connections coming through all working instances of the ingress controller, making the setting accurate across your cluster. This allows users to define precisely what amount of connections should reach the application services/pods. The number of running ingress controllers now does not impact this, which is especially important when services and their pods are highly sensitive to the number of connections they can receive.

The diagram below illustrates how the maxconn value is divided among multiple instances of the ingress controller.

New ALNP option

A new option, tls-alpn, is available for the ConfigMap resource. This allows you to set what the ALPN advertisement is when TLS is enabled. Its default value when not set remained the same: h2,http/1.1, which means prefer HTTP/2 but fall back to HTTP/1.1. For example, with this option you can disable HTTP/2. Simply use the value http/1.1 for this setting.

Implementation specific path type in ingress rules

The ingress controller supports all currently available versions of the Ingress API:

• extensions/v1beta1 (Required for Kubernetes < 1.14)
• networking.k8s.io/v1beta1 (Kubernetes < 1.19)
• networking.k8s.io/v1

While they share a lot of similar structure, some differences exist when adding the PathType attribute to an Ingress definition. To correctly handle all implementations, the path type is treated according to two rules: an unknown path type will be rejected and an empty path type with the v1beta1 API version will be set to ImplementationSpecific. This allows us to preserve behaviour for older resources, while at the same time use the new attributes that were introduced.

Multiarch Support

People run Kubernetes on many different platforms, and many of them are beginning to be quite popular. Recall that the processor architecture that Docker container targets must match the host operating system on which it runs. We have expanded availability to the Arm platform and now the ingress controller Docker image is available on these platforms:

• linux/386
• linux/amd64
• linux/arm/v6
• linux/arm/v7
• linux/arm64

Images on all platforms are pulled the same:

Here, a tag is the desired version of the controller that you want to install.

s6 Init system

We now use s6 overlay as a process supervisor inside the Docker container. The Docker image runs two processes simultaneously: the HAProxy load balancer and the ingress controller service; s6 allows it to do this in a more robust way. The s6-overlay provides proper PID 1 functionality so our services can concentrate on what they need to do.

Nightly builds

For users eager to try the latest features that won’t be available until the next release, nightly builds create Docker images that can be pulled with the docker pull command:

Or, edit the ingress controller’s deployment YAML file to use the nightly tag. A YAML file exists for deploying a Deployment or DaemonSet to your cluster.

External mode

Beyond deploying the ingress controller as a Docker container, with every release binaries for 12 different platforms are immediately available for download. This is needed in order to use the controller in external (out of cluster) mode.

These binaries are built automatically with goreleaser and are available on the project’s Github Releases page.

Contributions

We’d like to thank the code contributors who helped make this version possible!

Moemen MHEDHBI	CLEANUP BUILD DOC BUG FEATURE TEST REORG OPTIM
Ivan Matmati	FEATURE DOC BUG
Zlatko Bratkovic	CLEANUP FEATURE DOC BUG TEST REORG BUILD
Dinko Korunic	FEATURE BUILD
Robert Maticevic	BUILD
Nick Ramirez	DOC
Cristian Aldea	DOC FEATURE
Anton Troyanov	FEATURE
Fabiano Parente	BUG
Pasi Tuominen	BUG FEATURE
Kevin Ard	BUG
Toshokan	BUG

Conclusion

The biggest update is the introduction of CRDs. Version 1.7 uses the power of custom resource definitions to bring you more access to the features present in the underlying HAProxy engine. CRDs support better validation and control, even before committing changes to your Kubernetes cluster. These features strengthen the flexibility and security of your ingress solution.

Interested in learning more about the HAProxy Kubernetes Ingress Controller? Check out our Ingress Controller Documentation & don’t forget to Subscribe to our blog!