The U.S. government has cataloged over 1,700 active AI use cases, and nearly 90% of federal agencies are already using or planning to use AI. The European Commission has disclosed nearly 1,500 AI use cases across EU member states.
With over 3,200 combined AI use cases cataloged across the US and EU, public sector IT leaders face an identical roadblock: traditional application delivery controllers were not designed to parse or throttle Layer 7 LLM payloads, leading to backend GPU exhaustion.
A landmark 2025 OECD report analyzing 200 AI use cases across 11 government functions found that common barriers to scaling include skills gaps, data quality issues, outdated legacy IT systems, and weak measurement of return on investment. |
Generative AI in the public sector introduces demands that traditional architectures were never built for: sensitive data flowing through inference APIs and strict audit requirements.
If you're in the public sector, see how HAProxy One handles government application delivery across on-premises, sovereign cloud, and air-gapped environments, with commercial products available to US buyers through authorized partners on Federal and SLED contract vehicles.
How is AI applied in the public sector?
Government agencies use AI for specific, already-deployed tasks.
A few examples:
Fraud detection. The Treasury Department uses machine learning to detect fraud in real time, and recovered over $4 billion in fraudulent funds during fiscal year 2024. Carahsoft
Service chatbots. The IRS deployed a virtual assistant called "Ask IRS," built on Microsoft Azure AI, which handled over 3 million taxpayer questions in its first year and cut call center volume by 40%. SmartDev
Document processing. The General Services Administration uses AI to help review procurement documents, which frees up staff for more strategic work. SmartDev
Threat and anomaly detection, where models flag unusual network activity or suspicious transactions that a manual review would likely miss.
Agentic AI in the public sector
These examples share a pattern: a model flags something or answers something, and a person decides what happens next. Agentic AI removes that middle step. Instead of flagging an anomaly, an agentic system can investigate it. Instead of answering a question, it can complete the underlying task, such as processing a benefits application from start to finish with no human in the loop.
Securing sensitive data in AI inference APIs
Government AI handles some of the most sensitive data in existence: tax records, health histories, law enforcement intelligence, and defense communications. When that data moves through an inference API, it meets threats that ordinary web security was not designed to catch. In the EU, the General Data Protection Regulation (GDPR) raises the stakes further, requiring strict protection for any system that processes the personal data of EU citizens.
The endpoints themselves face a specific set of risks:
Prompt injection that manipulates a model into leaking data or behaving unpredictably.
Unauthorized access to inference endpoints, which exposes both model capabilities and the data sitting behind them.
Model extraction, where an attacker sends repeated probing queries to reverse-engineer proprietary model behavior.
Autonomous risk escalation: as agentic AI in the public sector spreads, a single compromised endpoint can trigger actions with no human in the loop.
Both the U.S. and the EU are answering with governance frameworks, including NIST's draft Cybersecurity Framework Profile for AI and the EU AI Act. Those frameworks set expectations; the delivery layer is where they actually get enforced.
HAProxy One application delivery platform enforces them through a web application and API protection (WAAP) stack. A web application firewall solution inspects requests and helps protect inference endpoints, achieving 99.65% balanced accuracy in an open source WAF benchmark. Meanwhile, bot management and protection identifies, classifies, and labels high-impact bot threats including application DDoS attacks, brute force attacks, web scrapers, and vulnerability scanners. For AI traffic specifically, an API gateway and AI gateway add rate limiting tied to API keys and token counts, which gives finer control than IP-based limits on their own.
HAProxy One runs consistently across on-premises, sovereign cloud, and air-gapped environments, giving government and public sector teams one security layer for AI services across the environments they operate in.
High availability for mission-critical AI services
Some government AI services cannot go dark. Disaster response coordination, real-time border monitoring, fraud detection across benefits systems, and emergency dispatch all fall into this category. When these systems stall, the cost is measured in public safety: in response times, and in people who can't reach the services they need.
Inference workloads make availability harder to guarantee. They are computationally heavy, and their latency swings with prompt complexity, model size, backend load, and concurrency. A delivery layer built for this kind of unpredictability keeps services responsive in a few concrete ways. Load balancing solutions spread inference requests across model replicas and continuously health-check the backends, rerouting traffic away from any instance that falters. HAProxy can run these health checks against specific endpoints, such as /health or /v1/models, to assess backend responsiveness and reroute traffic away from an overloaded GPU instance before requests start timing out. SSL/TLS offloading moves encryption work off the AI backend so latency stays low even under heavy load. Content-aware routing then sends different request types, such as chat completions and image generation, to the backend pools suited to them, so expensive GPU capacity goes where it is actually needed.
Latency compounds at this layer. When inference already adds hundreds of milliseconds, every fraction the delivery layer contributes counts. HAProxy is built for this: benchmarks showed HAProxy achieved over 2 million requests per second with TLS termination enabled, so the delivery layer adds negligible overhead even as inference traffic scales and demand spikes.
Modernizing legacy systems without a rip-and-replace
Public sector teams rarely get a clean slate. They run mainframe-era systems next to virtual machines and, increasingly, Kubernetes deployments. Older infrastructure predates modern AI workloads by decades, and replacing it wholesale is rarely realistic on a public budget or a public timeline.
HAProxy One deploys consistently across on-premises data centers, public clouds, sovereign clouds, and container environments, which lets it act as connective tissue between legacy systems and new AI services. A Kubernetes ingress controller routes traffic into containerized AI workloads while the same platform manages external load balancing for Kubernetes and the traditional backends beside them. Agencies modernize one service at a time, keeping central control and observability intact, and without locking themselves into a single cloud or compromising between jurisdictions.
Modernization works when the delivery layer stays constant, and the workloads behind it change at the agency's own pace.
AI governance in the public sector: observability obligations
AI governance in the public sector carries legal weight. In the U.S., OMB Memoranda M-25-21 and M-25-22 direct agencies to set AI strategies and to put safeguards around high-impact use cases, while updating how they buy AI. In the EU, the AI Act took force in August 2024 and classifies many public sector uses as "high-risk," which brings requirements for risk management, data governance, technical documentation, and fundamental rights impact assessments.
The two regimes approach the problem differently:
Jurisdiction | Approach | Core requirement |
United States | "High-impact" AI must meet minimum risk-management practices | Pre-deployment testing, ongoing monitoring, human oversight, and public transparency |
European Union | Four risk tiers, ranging from unacceptable down to minimal | Risk management system, data governance, technical documentation, and fundamental rights impact assessment |
These obligations are largely organizational, but several of them (ongoing monitoring, audit trails, and transparency) depend on infrastructure that can show its work. The HAProxy Fusion Control Plane aggregates logs, metrics, and security events from every node it manages, across on-premises, cloud, sovereign-cloud, and Kubernetes environments. Secure management traffic runs over mTLS, so the audit trail itself is protected end to end.
The OECD's guidance on trustworthy government AI points the same way, urging agencies to favor high-benefit, lower-risk applications while they build maturity. Most still lack the measurement to make those calls with confidence, which is exactly why infrastructure-level observability belongs in the design from day one, rather than something retrofitted after a deployment is already live.
Securing your agency's AI future
Every challenge described above has a known answer at the infrastructure layer. The real question for public sector IT teams is whether the platform under their AI services was built for this much load and this much scrutiny.
HAProxy One pairs the performance of the open source HAProxy core with a unified enterprise platform. The HAProxy Enterprise load balancer delivers high-performance load balancing and multi-layered security with token-aware rate limiting for AI traffic, and the HAProxy Fusion Control Plane centralizes management and observability across the estate. ML-enhanced threat intelligence already powers the detection algorithms in the HAProxy Enterprise WAF and Bot Management Module, and the HAProxy Edge application delivery network extends that intelligence to the network edge.
Reach out to discuss how HAProxy One can support your agency's AI infrastructure.
HAProxy's commercial products, including HAProxy Enterprise and HAProxy Fusion Control Plane, are available to public-sector buyers. In the United States, they can be procured through established Federal, state, and local contract vehicles.
Federal agencies can procure via authorized resellers or partners through SEWP V, ITES-SW2, GSA MAS, and DOD ESI BPA. State and local agencies can buy through TIPS, OMNIA Partners, OMNIA Partners ACS, NASPO ValuePoint, GSA Cooperative Purchasing, CA SLP, VITA, and CMAS. USAL is also available as a US contract vehicle for HAProxy's commercial products.
Public sector buyers outside the US can contact the HAProxy team to discuss procurement options for their region.
