PayPal runs six data centers, each with around 60,000 containers. Their 30,000 employees spin up nearly 10,000 test environments every day — roughly 6 to 10 every minute. Each environment requires three config updates: one to create the virtual service, and additional calls to configure and deploy the applications. Do the math and you get a staggering 30,000 config updates per day.
As a result of the scale involved and the agility developers expected, what began as a routing and load balancing challenge became an infrastructure automation challenge.The HAProxy One platform was the answer, with HAProxy Enterprise (the data plane) providing flexible and high-performance routing, and HAProxy Fusion (the control plane) discovering backends and automating configuration at scale.
The test environment routing problem is harder than it looks
Testing at PayPal is even trickier than one might expect. Developers need to test custom app versions against a backdrop of 3,000+ live applications. If a dev has a custom version of App A deployed, traffic needs to find it. If that version doesn't exist, traffic must fall back to the live version automatically.
This process is pretty straightforward with just one app. With a group of apps — all from different teams, at different stages, in different availability zones — it gets exponentially complicated.
Each test environment is essentially its own routing context. Virtual services group related apps together so traffic stays within the right environment. When an app isn't in a given group, traffic has to fall back intelligently instead of simply failing.
The routing requirements get more specific from there. PayPal needed the following:
Requirement | Detail |
A/B and canary deployments | Route just 1-2% of traffic to new app versions for production validation |
TLS termination at the VIP | Terminate TLS at HAProxy, use plain TCP internally |
Header rewrites per VIP | Redirect vanity URLs like |
Bi-directional routing | Preserve the original source IP, which Kubernetes Service objects normally mask |
Header, path, and port-based routing | User-configurable per environment |
The source IP issue in particular was a sticking point. Standard Kubernetes Service objects surface the node IP rather than the actual client IP, making it hard to trace traffic accurately. PayPal needed full visibility in both directions.
Building on the Map API
The architectural move that made this work at scale was treating routing rules as data, not configuration. This made it easier for developers to deploy new virtual services using PayPal's internal Genesis API layer, which orchestrates the full environment lifecycle.
In HAProxy Fusion Control Plane, the Map API stores routing rules in memory-resident maps. When PayPal needs to add a new test environment, a single API call updates the routing rules instantly — no config file rewrite, no service interruption. With 6 to 10 new environments spinning up per minute across 60,000 containers, this is the only architecture that keeps up. Changes take effect in milliseconds, and the system never falls behind its own workload.
Genesis calls the IPAM API to reserve a VIP, then calls the Map API to set up routing rules. Frontend configuration updates land in 10 to 15 seconds end-to-end — the full Genesis API sequence from IPAM reservation to routing rules in place.
HAProxy Enterprise load balancer's native support for header-based routing, TLS termination at the VIP, and source IP preservation handles the remaining requirements from PayPal's spec.
Service discovery does the backend work
The other half of the equation is backends. With 60,000+ containers across multiple clusters, manually managing backend pools isn't realistic. HAProxy Fusion's Kubernetes service discovery watches the cluster for new Service objects and automatically adds matching pods to the backend pool.
Instead of manually managing backend pools and configuration files, PayPal gets unified orchestration, observability, and dynamic service discovery from a single control plane. When a developer deploys an app, HAProxy Fusion discovers the new pods and brings them into rotation without any manual config changes.
PayPal runs four Kubernetes clusters. With service discovery supporting up to 12,000 Service objects per cluster, that covers 48,000 service objects across a single HAProxy Fusion deployment. Their four HAProxy Enterprise clusters are all managed through a single HAProxy Fusion control plane.
As Srivignessh Pacham, the senior software engineer who led this work, put it: "During deployment, you don't have to touch the HAProxy Enterprise cluster at all. HAProxy Fusion discovers those matching services and automatically adds them to the backends."
Scalability testing results
Before going to production, the Genesis team ran load and performance tests well beyond their daily requirements. The results:
Metric | Result |
Config update time | 10-15 seconds end-to-end |
IPAM API throughput | 50 virtual service calls per 2-minute window (5x their requirement) |
Map API throughput | 200-500 calls per 2-minute window; peaking at 1,000 updates per minute |
Service discovery | Stable well past the 12,000-backend target — tested up to 22,000 backends |
That last number is worth pausing on. The supported limit was 12,000 backends per HAProxy Fusion cluster. They pushed it to nearly double that and the system held. For a team planning ahead to production workloads, that headroom matters.
HAProxy Fusion’s Request Explorer also proved its value during testing. When something went wrong — an SSL/TLS handshake failure, a 500 error — the team could pinpoint exactly where in the request path the issue occurred. The Genesis API also lets developers run basic container introspection commands (top, ps) without navigating RBAC directly, routing them straight to the right container. With 10,000 distinct environments active at any time, reducing friction on every debugging touchpoint is essential.
Brief architectural overview
Each HAProxy Enterprise cluster is an active/standby pair. Genesis selects which cluster handles a given request. A single HAProxy Fusion control plane manages all four HAProxy Enterprise clusters, providing centralized monitoring and configuration.
The sequence for a new test environment:
Genesis API selects the target cluster
IPAM API reserves a Virtual IP
Map API updates routing rules in memory
Service discovery finds the Kubernetes pods and populates the backends
The developer goes from "code deployed" to "environment ready" in seconds.
PayPal's setup is a solid reference architecture for any team dealing with high-frequency and dynamic application deployment at scale. Map API-driven frontend routing paired with automated Kubernetes service discovery removes the manual overhead that makes environments like this hard to operate — and turns 30,000 daily config updates into just another Tuesday.
Watch the full presentation on PayPal Genesis from HAProxyConf 2025 for the complete walkthrough, including scale test data and architecture diagrams, or schedule a demo of HAProxy One.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
