Traditional HTTP follows a simple model: clients initiate connections to servers. But what happens when the server can't be reached directly due to NAT routers, firewalls, or network topology constraints? This fundamental networking challenge has driven the development of an innovative solution that turns the conventional client-server relationship on its head.
HAProxy developer Amaury Denoyelle introduces Reverse HTTP—a protocol that allows servers to initiate connections to a central gateway, which then reverses the connection flow to enable external client access. This breakthrough approach, implemented in HAProxy 2.9, leverages HTTP/2's multiplexing capabilities to create persistent tunnels that bypass traditional connectivity barriers. Rather than fighting against network restrictions, Reverse HTTP works with them by having services connect outbound to a gateway that becomes accessible to external clients.
This session explores the technical foundations of Reverse HTTP, from its basic principles to advanced configuration scenarios. You'll learn how connection reversal works, discover practical implementation strategies for security and scalability, and understand when this protocol offers significant advantages over traditional approaches. Whether you're dealing with complex network topologies, implementing zero-trust architectures, or managing distributed services, Reverse HTTP provides a powerful tool for solving connectivity challenges that seemed insurmountable with conventional HTTP.
Slide Deck
Here you can view the slides used in this presentation if you’d like a quick overview of what was shown during the talk.
Hi, thank you for joining me today. I'll present Reverse HTTP, a protocol and an HAProxy feature. My name is Amaury Denoyelle. I'm a developer working on HAProxy at HAProxy Technologies.
First, let's look at the problem we want to solve with this new feature.
Imagine you're at home and you receive a new IoT device that exposes its functionality through an HTTP service. You set it up and can control it at home. Everything works great, but now you want to access it from outside your home network – from your work office or anywhere on the internet.
This isn't as easy as it could be because your home service sits behind a NAT router. You have to configure this router through its clumsy web interface to route internet traffic to your service.
Developers face similar challenges when working with Docker containers. If you want to give access to your containers outside of your host machine, the process resembles NAT configuration. It's also not straightforward because you have to remember the IP addresses of different containers and which ports you want to relay.
Let's use HAProxy to solve these problems. In the case of home service hosting, HAProxy acts as a gateway on the Internet. Its job is to relay traffic so you can connect to the gateway to access your service. But we still haven't solved the NAT traversal problem.
This is where Reverse HTTP comes in. The home service connects first as a client to the gateway, bypassing the NAT router entirely. Once the connection between the service and the gateway is established, the connection flips - the gateway becomes the client, and the service becomes the server.
That's all it takes. Now, as a user, you can connect to your gateway and browse freely to your home service, completely bypassing the NAT firewall.
Reverse HTTP has many different applications. I've already covered the NAT scenario, but it's even more useful if your router implements CGNAT. In this case, you don't have complete control over which ports can be forwarded. Reverse HTTP may be the only solution to host a service at home. The situation is similar if you don't have a static IP address for your router. Reverse HTTP can solve this by using a central HAProxy gateway.
Reverse HTTP is also useful in development contexts. Imagine several developers working on a web application. These developers often want to deploy their application to test it outside their laptops. They might want to access it through a mobile phone to test the web rendering. We could provide a central gateway so each developer can connect freely when their application is ready for testing. Each developer can then access the web application via the gateway. Developers can connect and disconnect freely as their application becomes ready or not ready for testing.
Reverse HTTP is also useful in security contexts, particularly when the HAProxy gateway is installed in a DMZ environment. With Reverse HTTP, HAProxy doesn't initiate any connections on its own. The service is responsible for connecting first to HAProxy. This means there are no outgoing connections from the DMZ. As an administrator, you can add new firewall rules to ensure such traffic is blocked. This is better from a security standpoint because it ensures that even if an attacker gains access to one of the machines in the DMZ, they won't be able to use it as a vector for a relay attack.
Finally, Reverse HTTP particularly shines in dynamic environments. The configuration on the gateway doesn't need to specify endpoints or IP addresses for different services. This can already be achieved via dynamic servers, an HAProxy feature. However, with dynamic servers, you still have to connect to the gateway to activate new servers as they become available. With Reverse HTTP, no interaction is required on the gateway. Each server connects to the gateway and is instantly available without any activity required on the gateway. It's one step further on the dynamic aspect.
Let's review how the Reverse HTTP tunnel is established.
On the right, a developer is working on a web application on their laptop. We have already installed an HAProxy gateway, and there are clients on the left who may want to test the application. The application isn't ready yet, so the developer isn't connected to the gateway.
Now the developer is ready. They connect to the gateway, and the connection is flipped. The gateway becomes the client, and the dev station becomes the server. In the context of Reverse HTTP, we say that the development machine is the dialer. It's responsible for initiating the connection.
Once this process is done, it is announced, and clients can connect to the HAProxy gateway to access the developer's web application.
Here's a summary of the basic principles of Reverse HTTP.
This feature was introduced in HAProxy version 2.9, which was only about one and a half years ago. It's still experimental—not due to any functional limitations but because we expect it to evolve, particularly in the configuration aspect. There may be breaking changes in future HAProxy releases.
Reverse HTTP relies on the HTTP/2 protocol for the reversal of the connection. This is required because this protocol allows for multiplexing several client connections on the same backend connection.
Finally, the way Reverse HTTP is implemented in HAProxy is quite similar to an already existing IETF draft. However, as of today, HAProxy is currently the only known implementation of such a protocol. If you want to test Reverse HTTP, you must use HAProxy on both the gateway and the dialer sides. In this case, HAProxy will act as a sort of sidecar to expose the internal service through Reverse HTTP.
Here are two sample configurations to test Reverse HTTP. On the left is the configuration for the gateway, and on the right is the dialer.
First, let's focus on the gateway side. The gateway has two frontends: a public frontend for external users and a frontend dedicated to dialers. Note that this configuration isn't mandatory—you can arrange the frontends as you want—but it's better to separate them for clarity.
When external users connect to the public frontend fe-users, they want to access the server labeled as dev. This server uses an HTTP address, which means it cannot initiate a connection on its own. Since the dialer hasn't connected yet, external users will receive an HTTP 503 error code. There's no service available yet.
The dialer is expected to connect to the second frontend fe-rhttp. Once the connection is accepted, we do the reverse thanks to the new attach-srv rule. This rule first instructs HAProxy to reverse the connection, then the connection is inserted to the targeted server—the dev server—and inserted into its idle connection pool. The connection is now idle, and users can connect to the public gateway and use it to access the dialer and the service behind it.
Now for the dialer configuration. The dialer relies on a bind with an RHTTP address to open a connection. This bind specifies an internal backend to define the gateway's endpoint. It also specifies an endpoint parameter that instructs how many connections the dialer will open to the gateway.
We recommend using at least a number equivalent to the number of threads running on the gateway. If you use less, some users may still face a 503 error code. Finally, for the dialer configuration, we see at the bottom the service exposed through the dialer. In this case, it's on the loopback address on port 8080.
Let's discuss security with Reverse HTTP. The sample configuration so far has introduced no security. This means that any dialer could connect to a gateway, and external users may browse to an unknown dialer instance. As a gateway administrator, this is probably not what you want.
There's no built-in feature for this in Reverse HTTP. Instead, you simply have to rely on standard TLS. Here, the configuration has been updated, and each line on the gateway and the dialer will activate SSL.
You must also understand that the TLS handshake is performed before the connection reversal. On the gateway, the connection is still on the server side. To verify the dialer's identity, you have to activate verification to perform mutual TLS.
Now you can generate and use a certificate authority file on the gateway. You can derive multiple certificates from it and distribute them to the different dialers.
A particular aspect of Reverse HTTP is that a server using an HTTP address can actually be used to access different physical machines. This is one unique feature of Reverse HTTP. If these different machines host a similar service, it may not be a real problem. But suppose you want to use different services, where each dialer has a specific service, as an external user to the gateway. In that case, you probably want to specify which dialer you want to connect to.
The simplest solution would be to declare a new server line on the gateway, with each dialer associated with its own server. However, this isn't very scalable. A better, more dynamic approach simply relies on HAProxy's ability to differentiate idle connections in the same pool.
For connection reuse, we already ensure that clients don't reuse a connection that doesn't share a different set of specific parameters. In particular, one parameter we check on connection reuse is the SSL SNI. Recently, the SSL-SNI reuse code has been reworked and made more general.
We have a new parameter associated with idle connections, called name. This parameter is more flexible, and we can use any key to identify the connection in a pool. Now, let's apply that to the Reverse HTTP case.
You have to perform this in two steps. First, on the attach-srv rule, we now use the name parameter to specify how the connection will be associated with a parameter. In the example here, each connection, when reversed and inserted in the connection pool, will be associated with the certificate's common name that the dialer presents. In fact, the name parameter can use a variety of HAProxy features, so you can use the parameter you want.
Now, the second step. As an external gateway user, you want to specify which connection from the pool you want to use, reusing the same key used for the attach-srv rule. This is the purpose of the pool-conn-name keyword. In this case, we're using the standard HTTP host header. In fact, this process is quite similar to standard TLS. When you use your browser and connect to a TLS server, the browser checks the common name of the certificate presented by the server and compares it with the host you're trying to access.
We have introduced new features for Reverse HTTP for the newest HAProxy 3.2 versions.
The first one is the check-reuse-pool keyword. This keyword can be added to the server line. It will instruct that checks are performed by reusing connections instead of creating a new connection. For Reverse HTTP, this is mandatory if you want to perform checks because, by definition, these servers are unable to connect by themselves. But this keyword is also useful outside of the Reverse HTTP context. For example, if you're using TLS on your server and you don't want to perform a full handshake for every check, you can use this new keyword.
The other keyword is idle-ping. Idle ping can be activated on the bind and server lines. It performs a simple check to detect if the connection is still active. It relies on the underlying multiplexer support. For now, only HTTP/2 implements a proper idle ping mechanism.
For Reverse HTTP, idle-ping is quite useful. For example, on the dialer, you can activate it on the bind line. This will allow the dialer to check the liveliness of the open connection periodically. If one connection is inactive, the dialer can reopen the connection to the gateway. Also, idle-ping is useful to ensure that connections won't be removed due to inactivity timeout on the gateway or the dialer side.
As stated earlier, Reverse HTTP is still subject to configuration evolution. This is why it's still labeled as experimental. We aim to simplify the configuration and go further with the dynamic aspect of it. We know that some users have already started to use it and deploy it for their own usage. We also use it at HAProxy Technologies for development ease of use.
Finally, we hope Reverse HTTP may gain some traction outside of HAProxy. If this is the case, we could imagine that other server software will start implementing it so that we won't need HAProxy on the dialer side anymore.
And in fact, you can also help by implementing Reverse HTTP in your favorite server language of choice. Please do not hesitate to contact us with any thoughts about it or any other interrogation.
Thank you for your attention, and have a great conference.
