This post will be updated over the next several days.
Recently, a Remote Code Execution vulnerability was discovered in the Apache Log4J library. This vulnerability, which is tracked in CVE-2021-44228, dubbed Log4Shell, allows attackers to execute arbitrary code on affected systems.
While HAProxy, HAProxy Enterprise, HAProxy ALOHA, and other products within the HAProxy Technologies portfolio are not impacted by this (they do not use the Log4J library at all), you can use them to block the attack.
These mitigation techniques we describe are subject to change as attackers come up with new evasion methods, new false positives are discovered, or better ways to handle this are found. When the rules change this blog post will be updated with a changelog added to the top. While these rules have been tested, it is possible that there are false positives or false negatives still remaining, so for sensitive environments we recommend testing these rules in your environment (such as replacing the http-request deny rules with an http-request capture rule to log violations rather than deny them outright). Use HAProxy or HAProxy Enterprise 1.8 or newer, or HAProxy ALOHA 10.5 or newer, to use the syntax we describe.
Regardless of these protections, if you are using Log4j in your environment you should update to 2.16.0 or newer from their download page.
If your applications log (or might log) information in the post body you might want to consider expanding the filtering of the post body. Some applications log user names or transactions id’s in the post body and should have req.body filtered, due to false-positive risk with there being no json_parse decoder we don’t include it by default (nor does ModSecurity CRS), but it can be enabled easily if it fits your environment (and you can monitor/tolerate false positives). This can be done by adding more deny/acl lines like ‘log4shell_form’ to decode/scan specific content types if using ACL’s or by adding REQUEST_BODY to the match zones on the SecRule line if using ModSecurity.
To block the attack, add the following ACLs, which are an adaptation of the ModSecurity solution described later, to a frontend or listen section within your load balancer configuration:
Alternatively, enable the ModSecurity WAF module:
Find the following line in the file /etc/hapee-2.4/modsec.rules.d/lb-modsecurity.conf:
Or find the following line in the file /app/security/etc/sec-offloader/modsecurity.load in HAProxy ALOHA :
Then add the following rules directly after:
These rules can be found on the Log4j blog post from coreruleset.org. If this is the first time you are enabling the ModSecurity WAF, we recommend running it in detection-only mode at first, to log malicious requests, but not immediately block them.
With HAProxy Enterprise, you can also block the attack by enabling the Advanced WAF:
Then update your ruleset files by updating to the latest hapee-2.4r1-lb-wafadvanced package (or using the corresponding package for your version of HAProxy Enterprise). Follow the documentation for how to set thresholds for blocking threats.
If this is the first time you are enabling the Advanced WAF, we recommend running it in learning mode at first, to log malicious requests, but not immediately block them.
To test if the mitigations you have configured are working correctly, try the following curl command:
If the blocking rules are working correctly, that request will be denied.
Changelog:
- 12/15 9:30 AM EST: Updated ModSecurity rule based on CRS guidance
- 12/15 9:30 AM EST: Fixed incorrect filename for HAProxy ALOHA
- 12/15 11:00 AM EST: Updated ACL example to use simplified rules
- 12/15 5:15 PM EST: Updated ModSecurity and ACL rules based on CRS guidance
- 12/17 3:41 PM EST: Added guidance on post body filtering
- 12/20 11:00 AM EST: Updated ModSecurity targets per CRS guidance
Hi,
On Aloha v7.5.3, we have this error
error detected while parsing ACL ‘log4shell’ : unknown fetch method ‘req.hdrs’ in ACL expression ‘req.hdrs’
If you are on an older version, then the “req.hdrs” fetch method may not exist, but has been available in the last several versions.
On older verions just use req.hdr() instead of req.hdrs. Same functionality.
Lukas Wolf, technically, req.hdrs is not the same as red.hdr(). req.hdrs returns all the headers with the values, while req.hdr(User-Agent) returns only the value of the User-Agent header.
From my understanding calling ‘req.hdr()’ literally without specifying a certain header name does return all headers?!
So if you have a rule in phase 2 like in these examples, it take take action/make decisions on EITHER phase 1 (request headers, etc.) or phase 2 (body) data?
Hi Lou, here’s a resource that describes the processing phases of ModSecurity: https://malware.expert/modsecurity/processing-phases-modsecurity/
Hi Nick
Thanks for replying. I was familiar with the phases, but I just thought to block something from data in the request headers, you needed a phase 1 rule and to block something triggered in the body, you needed a phase 2 rule.
Thanks for linking our blog at coreruleset.org. Given you are also copying our rules, you may want to make sure you also copy the continuous updates to said rules.
Thank you Christian, we have updated the rule now based on the coreruleset.org guidance.
But when users uses older versions (1.5/18), how can they then protect against the log4j attack? Because upgrading is not so fast as people wish;-).
You may be able to use a combination of the older fetch methods, but after a certain time, we don’t support older versions and recommend you upgrade.
Haproxy before 1.7 doesn’t do a full url inspection.. there is a character (buffer) limit so I would recommend to upgrade to at least 1.7 to properly enable the `option http-buffer-request` option and properly mitigate these CVE’s .
Please Note,, an oversize uri could bypass these rules altogether on older versions of haproxy… main 1.5 which is a default in many linux distros
I’ve found the upgrade to be relatively painless. You may see some warning on the sorted directives, but for the most part, it is backwards compatible.
BTW. Huge thanks for these acls.
It looks like the ACL regex for HAProxy is incorrect and missing a closing bracket at the end.
Good catch. We have fixed this now by adding the closing bracket.
Hi, is that the code changed recently (in the last 1 day)? Because I can before there wer more lines of code now just 6?
Yes, the rules have been modified and simplified.
Is there a way to block POST requests with Content Type multipart/form-data? It can be tested with: curl -v -X POST -F ‘test=the_bad_word_here’ some_url
For the best protection of form-data, we recommend the WAF because it understands, validates, and decodes sections prior to scanning.
curl -X POST \
https://XXXXXXX \
-H ‘authorization: Bearer XXXXXXX’ \
-H ‘cache-control: no-cache’ \
-H ‘content-type: application/json’ \
-H ‘postman-token: c29fa9d0-fbdd-cbe5-4a67-d968e9a9e4ee’ \
-d ‘{
“transaction_id”:”${jndi:ldap://127.0.0.1:1099/ouuuuuubj”,
“transaction_date”:”2020-07-23″
}’
it still cant block
In that case it is because the content-type of application/json isn’t in our ACL’s (if you are using that) or that REQUEST_BODY isn’t in the ModSecurity rule (if using that). There is an increased risk of false positives when scanning the body, and since a lot of applications don’t log anything from the post body it often isn’t worth it. That is a good point though and I’ve added a paragraph to the blog post explaining how to expand coverage of the request body if the benefits outweigh the false positive risks for a given environment.
“http-request deny” immediately returns an HTTP 403 error to the client, making it easier for the attacker to try other ways around the defenses.
Using “http-request silent-drop” will leave the attacker waiting for a response for a long time, reducing the efficiency of your bots.
Wouldn’t it be better to use “silent-drop” instead of “deny”?
“http-request deny” immediately returns an HTTP 403 error to the client, making it easier for the attacker to try other ways around the defenses
Using “http-request silent-drop” will leave the attacker waiting for a response for a long time, reducing the efficiency of your bots
Wouldn’t it be better to use “silent-drop” instead of “deny”?
silent-drop will cause problems with stateful firewalls (not removing the connection from their connection table) and shouldn’t be used without thinking about it for a specific environment. In general, attackers will likely guess that filtering is at play if one string works but a log4j string fails in some way; a 404 might be a bit better, but in the event of a false-positive also runs the risk of increasing confusion.
Hi Nick,
could you elaborate on the part of the string you are trying to capture, i.e. the bad parts with some example strings?
While correct syntax and will work on some test strings, the regex might not be crafted correctly.
Thanks for sharing this too.
Dave
HAProxy 2.3.4 seems to complain that the current regex is invalid. I can’t seem to figure out what it doesn’t like.
[/etc/haproxy/haproxy.cfg:151] : error detected while parsing ACL ‘log4shell’ : regex ‘…redacted…’ is invalid.
[/etc/haproxy/haproxy.cfg:152] : error detected while parsing ACL ‘log4shell’ : regex ‘…redacted…’ is invalid.
Any luck with this?
How can i handle with this problem in mode tcp?
That depends on the protocol, if it is HTTP or a similar client-speaks-first protocol you can apply the regex to the buffer (though if an attacker uses url encoding, expect 100 continue or similar they can easily bypass this):
tcp-request content reject if { req.payload(0,0) -i -m reg (?:\${[^}]{0,4}\${|\${(?:jndi|ctx)) }Depending on the protocol and how acceptable false positives are for your use-case you might wish to block on a more broad string like jndi:.