On Thursday, December 9th, 2021, a critical zero-day exploit was reported in the widely used log4j2 library. The exploit did not affect the Hedera network. Out of an abundance of caution, the Hedera team addressed the issue on Friday with the highest priority, and was able to do so without any downtime or disruption to the network.
The library log4j2 contains a powerful feature called "lookups" that can be used to include system information in the logs automatically. A lookup command can download and inject remote code into the Java Virtual Machine (JVM) process, enabling an attacker to execute remote code and gain access to the system. Any application that takes user supplied code and logs it can be exploited if the user sends a specially formatted string that contains lookup commands within it.
For an example, check out the Impact section of this blog post by Randori. The diagram in that post illustrates how an attacker can leverage this vulnerability. In the example, the specially formatted string is “${jndi:ldap://127.0.0.1/a}”.
The version of log4j2 used by the Hedera network (2.13.x) is susceptible to this exploit, but only if user-supplied code is logged.
There are three ways to mitigate this attack (other than never logging user input in the first place).
First, upgrade the library to a version greater than or equal to 2.15.0. Upgrading to this library requires thorough code analysis and a full test/debug code cycle, which takes some time and requires downtime on the network. We opted not to do this for Friday December 10th because it would take too long and rushing that process could expose the network to unnecessary risk.
Second, set the `log4j2.formatMsgNoLookups` flag to true. This effectively disables the feature without requiring a full code/test/debug cycle. Setting this flag would mean either downtime on the network or the node operators performing a rolling upgrade. A rolling upgrade would mean taking down nodes in small groups so as not to disrupt consensus and restart them, waiting for them to fully reconnect.
The third option, and what was done, update the log4j2 configuration to disable lookups within the configuration file itself. Changing the configuration can be done without downtime and without rolling upgrades. Configuration file changes are automatically reapplied by log4j2 every 10 minutes.
For mirror nodes, the log4j2.formatMsgNoLookups flag was set to true in all environments and a rolling upgrade was performed. Then the version for log4j2 was upgraded, and a new mirror node version was released and rolled out to all environments.
The validated changes were taken up by modifying the log configuration in a way that is visible in our log files. The change was applied to all public Hedera networks (preview networks, public Testnet, and Mainnet).
The Hedera team is now working on reviewing log4j2 version 2.16.0 for deployment in a future release.
Mirror Node Operators: The Hedera team backported the lo4j2 fix to previous releases of the mirror node, from v0.42 to v0.46. It is recommended that mirror node operators upgrade as soon as possible. Access the updated releases at: https://github.com/hashgraph/hedera-mirror-node/releases
Swirlds SDK Users: Options two and three in the previous section would fix the issue for users of the Swirlds SDK.
Option two (setting the log4j2.formatMsgNoLookups flag to true) helps those unable to upgrade log4j2 to version 2.16.0 or greater. In releases 2.10 or greater, this option can be implemented as follows when starting the JVM:
add -Dlog4j2.formatMsgNoLookups=true
For releases older than 2.10, check out the recommendations in the Randori blog post.
Option three (updating the log4j2 configuration to disable lookups) can be implemented according to the Apache documentation: https://logging.apache.org/log4j/2.x/manual/layouts.html#:~:text=Outputs%20the%20application,JAnsi%2C%20see%20configuration
Hedera Client SDK Users: These users are not affected, so no action is needed.