Agentic Security at HAProxy, and the struggles of security politics when you're not special
This is a look at my own personal experience, and does not reflect the thoughts of everyone at the company or in the community.
I'm the lead developer of HAProxy and a member of the Linux Kernel security team. Over the last few months, I've seen hundreds of AI-assisted "security reports" almost always following the same pattern: someone submits "here's a bug" with no reproducer and no patch. It has just become routine to deal with incomplete reports because the reporter often has no idea what the LLM thinks it found.
Yet this is not an LLM problem at all. A few months ago, a trusted colleague who had access to Anthropic’s most powerful model, Mythos, used it to analyze HAProxy. Crucially, they took the time to validate the findings and write patches (thanks for that, by the way). I was genuinely impressed by the depth and complexity of a few of the bugs this model found, and wondered whether they would ever have been found without such tools. That experience gave me confidence in the model's actual capabilities and the results when paired with human expertise.
Reports we can't use, a model we can't get
To help address the flood of low-value AI reports, I documented the threat models for both Linux Kernel and HAProxy so LLMs could better calibrate severity and stop wasting everyone's time on issues that either can't happen or have no real-world impact. This has noticeably reduced the rate of boring reports on both projects (not enough yet on Linux, where some researchers are still focused on outdated versions lacking the documentation updates).
Around the same time, the Linux Foundation provided me with a six-month subscription to Claude Max with 20x usage. It took me a little while to adapt to working in agentic mode, but that’s OK now. The product itself is impressive, genuinely powerful, and clearly still improving. Its ability to work within a very large context window that covers the entire code area under scrutiny makes it feel almost as if it's seeing through walls. Unfortunately, that access didn't include Mythos, which remains subject to additional access restrictions.
For instance, access to models with true cyber “defense” capabilities has become draconian unless you know the “right” people. Even though I completed Anthropic's Cyber Verification Program (CVP) hoping it would solve this, it still grants no access to Mythos. I reached out to contacts at Anthropic requesting access; the request was apparently deemed legitimate, but still nothing came of it. Anthropic continues to mature its program, but each time they change the process, it means starting over with a greater administrative burden, with no certainty of receiving access for legitimate defensive work. I was told the same wasn’t much better for OpenAI’s Daybreak program, where our Security team had been rejected twice in recent months without explanation prior to their CVP updates.
The access situation has changed a bit and is now self-service from Anthropic, though with significant barriers of entry across their self-defined access tiers. The most capable “Mythos”- based models (“Red” access) require verified security researcher status, which apparently neither I nor anyone on our team can qualify for. The lower tiers require a tedious identity approval process geared towards corporations (I still couldn’t fulfill it) and only grant moderately capable models that refuse important defensive tasks such as exploit reproduction and other defensive tasks.
Instead, we eventually obtained two batches of Mythos-generated reports about HAProxy, each containing 30 to 50 entries, indicating that others had already been granted access without our knowledge (except for that first colleague). It took extraordinary effort to get them, and that was nothing compared to the effort required to work through them.
Those reports were painfully unusable by developers, full of verbiage crammed into super-long single-line paragraphs with no code excerpts, no clear explanation of the issue, let alone any reproducer or fix. They didn't even include a commit ID for the build they tested, so we had to guess by trial and error what version had been tested. This is very frustrating because the model has surfaced a finding with high confidence, but it’s not very willing to communicate efficiently, since it wasn't asked to. It makes one really doubt that Anthropic knows how developers work and what they need.
Since we didn’t have access to the model, we adapted our workflow by tasking its little brother, Opus 4.8 (what we had available at the time) to decode these reports and propose fixes. It worked reasonably well, though the absence of reproduction code made it difficult to distinguish real defects from false positives, and thus validate the relevance and effectiveness of the proposed patches. Given that all the reasoning context was lost before the report was shared, Opus couldn’t reconstruct all the interactions or everything Mythos observed. Still, this streamlined workflow allowed us to work more efficiently and merge about one-third of the proposed fixes after reworking them to better align with our standards, preferences, and long-term goals.
Safeguards fire at the wrong moment
Following that, the release of Fable 5 initially seemed promising, but it introduced nasty safeguards that render it unsuitable for low-level development where performance and security matter. They trigger constantly. There's no realistic scenario in which you could scan your code for security issues, let alone read a security report with Fable; it instantly refuses and demands downgrading to Opus 4.8. Apparently, the word "bug" in the prompt combined with reading C code is enough to instantly flag everything as cybersecurity work that shouldn't proceed. To date, I still have no idea whether anyone has successfully used this model, or for what purpose. Certainly not low-level development, at least.
The subsequent arrival of Opus 5, followed by 5.5, delivered impressive improvements. They really feel like tools designed to help the developer, almost like a skillful coworker. You can delegate long and complex tasks without worrying what you’ll get at the end; they’re well organized, thoughtful, and even tend to anticipate some requests by spontaneously proposing nice extras. The safeguards trigger much less often than Fable but still too frequently to be reliable for developers writing secure code that needs to be verified (otherwise it’s not secure, is it?). At least they can find bugs now, and occasionally produce a working reproducer. But often when you already have a reproducer, they blatantly refuse to study the root cause of the bug!
When refusal happens, the approach that seems to work most of the time is to kill the session and restart from scratch, trashing whatever upset the model in its context, hoping the different initial random seed will steer it away from emitting the same token sequence that triggered the same safeguard at the cost of more tokens spent. Sometimes it takes 3-4 attempts or I have to accept a downgrade to Opus 4.8 and wait for ages for it to figure out a reproducer. Overall, the capabilities of these models remain pretty compelling despite this annoyance.
At the time, I didn't notice any meaningful difference in blocking rates between my work account (which didn’t yet have any security access) and the validated CVP account I had through the Linux Foundation, leaving me unconvinced that the latter added any value.
Access goes to the people doing the least work
Then came this odd turn: Anthropic published what amounted to an advertisement for their strongest competitor, GLM 5.3. In fairness, it wasn't meant as an ad; they were complaining about how competitors avoid annoying users with safeguards like Anthropic does in their long war on open weights which no longer fools anyone. But for those dealing with stubborn Opus or Fable refusals all day long, this lands very differently: that's exactly what they want! It made me curious enough to test GLM 5.3 myself. True, it does find bugs, runs faster, and costs less. On a first pass, it found roughly half the bug candidates Opus 5.5 found, but it also caught at least one real bug that Opus missed. Probably just need to run it twice, and at a fraction of the cost too.
Last month, Anthropic announced (with a surprising greeting image indicating that their marketing people were not on the internet before 2004) an "improved Cyber Verification Program" (CVP), which still doesn't fix anything, and is probably even making things worse for Open Source developers and maintainers without access to corporate resources. Individuals can only access the "defense" tier, which, according to Anthropic’s own graph scores, performs extremely poorly at discovering bugs and, at least in my own testing, appears identical to what I currently have on my personal account. Higher tiers are reserved for security professionals, meaning people who make a business of harassing open-source maintainers with pointless, unexploitable reports and then demanding CVEs to measure their relevance against competitors.
See the problem? They're giving more access to those causing harm while offering nothing better to the people doing the actual work of reading long and complicated reports, reproducing issues, validating them, issuing fixes, authoring descriptive commit messages and porting to maintenance branches. Anthropic is magnifying the burden on the Open Source community instead of helping.
Open Source maintainers should get prepared for an influx of reports, some good, some bad, but forcing them to do even more painful bug-fixing work for free while others are building their company’s reputation on top of their efforts.
What developers actually need
Let me clarify what most developers actually need, as confirmed through discussions with several other OSS maintainers:
Scan their code themselves to find bug candidates in specific areas
For each bug, try hard to write a reproducer proving the bug is reachable in existing code without cheating by modifying it. A good reproducer is one you can reuse later, when backporting fixes to maintenance branches, to verify that the fix was needed and worked there as well.
For each bug, write one or preferably several fix candidates from the same session's context, while that context is still rich enough to understand all the interactions that led to the bug being flagged, and explain why those fixes would durably work.
Eliminate false positives identified by the steps above (writing fixes really sorts them out).
Write tests to ensure the fix didn't introduce regressions, and check for other potential bugs in the same class.
What most developers do not need:
To write weaponized exploits (e.g., shellcode injection)
To penetrate running systems and hop from component to component
To attempt privilege escalation
To exfiltrate confidential data
To affect data integrity
To spend time triaging and reviewing totally disorganized reports about arbitrary code locations that are difficult to organize and dispatch to maintainers
The current poorly tuned (or designed) safeguards stop maintainers at step one, while all the other capabilities remain freely available to paid customers who continue making a business of selling exploits and pissing everyone off. I think that at the very least, the safeguards shouldn’t trigger as long as the model is working on code whose source is provided. What’s the point of writing a candidate bug fix, compiling to test it, and suddenly claiming “I’m not allowed to attack this code”?
In addition, even denying offensive tasks is short-sighted. The main point of software is that it receives fixes. We’re all currently undergoing a painful period where many bugs are reported, but at the same time software is getting fixed, so eventually there will be nothing left to be attacked. It’s not like genetics, where you cannot decide to change your genome once a disease is detected.
What would actually help
A better approach? Consider this: GitHub hosts 225 million developers across 800 million projects. A small-fee subscription that lets these developers work on public projects without the absurd restrictions I've described above would be enormously helpful and highly profitable for any AI vendor offering it. The LLM could accept to work on a local repository that didn’t diverge too much from an authorized public one.
Personally, I'd pay $20/month for this. And honestly, it wouldn't cost vendors much: my current Claude Max 20x account only once reached a two-digit weekly usage ratio after two full days torturing the HAProxy code to make it expel unknown bugs. Security reviews are remarkably cheap with tokens because most of the time is spent building and running local reproducers, not reading or writing code. Note that Microsoft already owns GitHub, owns Copilot, has access to these accounts, and has built some models themselves. Maybe they could see the benefit of jumping into this opportunity?
Another hope lies elsewhere. After several months, it's become clear that Anthropic doesn't understand how developers actually work (does this indicate that their own teams no longer consist of real developers but only "vibe coders"?). Perhaps we should put our hopes on other vendors instead. I don't know whether OpenAI's models are similarly unusable for this specific task that really determines what the security of tomorrow’s Internet will look like.
At work, I’ve been using Qwen3.8-Flash-Next with Strata on a local GPU with very satisfying results and acceptable speeds. Recently, Mistral released its new Large 4 model, with a clear focus on cybersecurity effectiveness. Let's hope it's more capable and better oriented toward what users actually need rather than the vendor's biased beliefs about what developers should be doing. A first test showed it was at least on par with GLM on this point, but, subjectively, between GLM and Opus 5.5.
In the meantime, I'll keep using Opus 5.5 when it's willing to cooperate until my subscription expires next month, at least to validate sensitive changes in HAProxy's codebase and write regression tests when it doesn't refuse mid-task. In parallel, I'm familiarizing myself with GLM 5.3 at work to figure out if it’s possible to help it find more complex bugs (and I cannot express how grateful I am to Anthropic for this hint).
It's clear that nothing will convince me to renew my personal Claude subscription with its current limitations; it is extremely frustrating to use with code designed to be secure (but it is excellent where security is not a concern). I will definitely miss it for certain tasks, as it's extremely powerful, but it's too stubborn when attempting serious work that it obviously can do, but simply refuses to. Of course, if Anthropic would just take the time to listen to what users actually need without any unwelcome ideological bias, this could be far easier for everyone involved.
And then there's the CLA
Immediately after writing a draft of this article, I was notified that Anthropic had opened a new OSS scanner project. I thought, “Great, finally some progress”. Of course it’s far from being perfect, and the Curl project’s experience confirms this; it still doesn’t permit developers to control the model and make it dig hard into specific areas or explore different approaches for a given bug, but that could be seen as progress in the right direction. That was until I read the CLA they require every developer to sign before participating in the program. In short, everything is concentrated in sections 2 (“Grant of Copyright License”), 3 (“Grant of Patent License”), and 4 (“Legal Entitlement”). Section 2 says:
You hereby grant to the Company and to recipients of software distributed by the Company a jperpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.
Section 3 says the same about patents, and Section 4 says, “You represent that you are legally entitled to grant the above license”.
How can the copyright thing not scare away any Open Source developer? It’s basically saying they can do whatever they want with the software submitted to the scanner. That might likely be due to their intended use of the software to train their models, but regardless, allowing them to turn it into anything can be a serious concern. There are many open-source licenses with different properties for a reason: developers don’t all have the same expectations about how their work will be used.
Point 3 is going to scare enterprises; it’s probably on purpose, to make sure they either give up their patents or use a paid model instead. And point 4 raises a problem in Open Source, where it’s far from certain that the person signing the CLA has the legal right to cede other contributors’ rights to them, as it resembles a unilateral license change made by a single person. Regardless, it looks like several hundred developers agreed to sign that thing.
Did they get legal advice before doing this?
Where this leaves us - better code, slower features
For now, I'll get back to reviewing the list of AI-generated security reports Opus and GLM produced. We have around 300-400 pending, with 2-3 that look serious at first glance (that ratio is not unusual; what matters is the total effort it takes to iron these ones out). And since we're interested in fixing any bug that could affect long-term stability, not just spectacular vulnerabilities that enhance a researcher’s curriculum vitae or just serve to promote a 3-week-old cybersecurity startup, we have to devote time, effort, and mental preparation to review all of them. Anthropic’s data seems to show the same, where only 0.5% of 7,300 vulnerabilities have actually been prioritized and fixed by the world’s open source maintainers.
We’ll see how all of this evolves. The positive point is that we now have even fewer bugs in HAProxy than older versions used to have, and that remains a net gain for our users; however, that effort definitely impacts the speed at which we progress on new features, which is why it would be nice to see this situation change.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.