Log4Shell: The Big Picture

So much for a quiet holiday season: CVE-2021-44228 (aka Log4Shell) may well be the most impactful vulnerability we've seen in years. Let's look at what makes this vulnerability special, how multiple factors came together to create the perfect cybersecurity storm, and some things we should get done when the storm passes.

My goal here isn't a technical walkthrough: Others have done that very well, so I'd like to go up a level and look at why this is such a tricky vulnerability and why the industry response has been good, but not great.

The Perfect Storm
Log4Shell has some particularly unpleasant properties, but what really stands out to me is that in many cases exploitation can be as easy as convincing a vulnerable service to log a piece of attacker-controlled data. Moreover, because the vulnerability is tied to data, it can go wherever the data goes, so the exploited system is not always Internet-facing — it can exist deep in cloud infrastructure. The second we had a fully working exploit, we realized that the blast radius of Log4Shell would be huge.

This perfect storm is a result of the following factors combining into one vulnerability:

Log4j is a common component, and frequently no privileged access is required to exploit a vulnerable system.
Externally, there is no 100% reliable way to know you're safe. You can know you're exploitable (that's how most of the current scanners work — launch the first stage of an exploit), but the only way to be sure you are safe is to patch all vulnerable Log4j instances everywhere. Given that this component is used in many back-end systems (such as Elastic and Logstash), that covers not just code you wrote but also packages you use.
The attacker only needs to be able to get the victim system to write something to a log … and there are myriad ways this can happen. Yes, it can be as simple as tainting the "User Agent" field in a request header — but it could be as complex as tainting a form field or, worse yet, an API parameter in just the right way.
There can be significant lag between launching a probe and receiving a result — this lag can lead to detection failures, where a vulnerable system is incorrectly marked as "safe."

When you add these things together, you get to where we are today: Things are rather a mess and likely to remain so for some time.

Friendly Fire and a False Sense of Security
As if the vulnerability itself wasn't enough, we should recognize the self-inflicted harm and the misguided sense of security that add to the context of Log4Shell.

One example of a well-intentioned but double-edged action is when third parties add rules that filter out strings that are closely associated with Log4j exploitation. On the one hand, that's great: It makes life harder for attackers. On the other, it makes the results of external scanning less reliable, and a defender may be unaware of how their site is being filtered or how good that filtering actually is.

For example, Praetorian's researchers built a scanner over the weekend that allows us to crawl customers sites, hunting for this vulnerability. Alas, various filters in front of the target means that we know there are requests that didn't make it through, blunting our ability to reliably probe for all instances. While we're pretty adept at manually bypassing rules like this, right now we simply don't have time for that kind of bespoke work … and so defenses in place have become an impediment to discovery. The upshot — especially if you're using a home-brew scanner — is that you may think you're fine, but you're not. You have to make sure the unaltered payload is actually landing on your machine.

A second problem comes from scanners in the market today that are of questionable quality. Such scanners may detect a few vulnerable instances, but their coverage is limited. A scan with such a scanner may produce a clean bill of health and false sense of security, when in reality, your attack surface is still exploitable.

Because of that, it's really important for vendors to be upfront about the limitations of their tooling. While Praetorian's scanner incorporated every trick in the book to have the highest possible detection rate, we are clear with customers that if we don't see the vulnerability, it doesn't mean that it's not there; it may just mean it's harder to find. External scanning has its limitations, and the only way to be sure is to go through your source code — and all the components you use — and root out vulnerable versions. My gut tells me we'll be seeing this vulnerability in custom security assessments for months or years to come. Why? Because I think we, as an industry, haven't always been as clear about our limitations as we could or perhaps should have been.

What Next?
The next few weeks will be spent digging out, but while you work on that, keep in mind that Log4Shell won't be the last vulnerability we'll ever deal with at this scale. As a community, we should be thinking about how to improve interoperability and response industrywide. There's a significant need for clearer communication, and some kind of coordination across defenders. Although the aftermath of Log4Shell will linger, we will get through it. However, shame on us if we don't start preparing for the next event today.

That means knowing where your assets are, knowing what they're running, and knowing what they talk to. No small task, but that storm is coming — be ready!

Results for “"picturea"”

Log4Shell: The Big Picture