ADYEN N.V. 
It has been quite a week for the Java Ecosystem and the Open Source communities: on December 9, 2021, a critical vulnerability was discovered in the popular Java Logging framework Log4j (version 2).
In this blog post I am going to talk about the Log4j vulnerability and how Adyen responded; I will share some of the actions, mitigation factors and best coding practices from our Security team and experts in the domain.
Reported to the Apache Foundation by the Alibaba Cloud Security team, the Log4j vulnerability has initiated a global race to understand exposure, apply a fix as soon as possible, and evaluate impact across the wide number of systems and software applications that somehow rely (or include) the library.
The vulnerability, being so severe and ubiquitous, has been given its own name (Log4Shell), something that happened in the past for other notorious security threats (https://shellsharks.com/designer-vulnerabilities).
A malicious attacker can exploit a Log4j security flaw triggering Remote Code Execution (RCE), extracting sensitive data or causing a denial of service, typically providing certain input strings that are eventually logged by the application using Log4j.
Leveraging on JNDI capabilities a remote class can be injected into the server process and executed with a very undesirable outcome.
Java is a pillar of our core platform and several services are running on top of it - therefore we have taken the incident very seriously. People, processes and tools are our best assets and they have again delivered: Let me share some insights of what happened and what we have done.
Soon after the news started circulating, our teams were already exchanging messages, performing an initial assessment and discussing the first steps and actions. Although we have formal channels for managing security alerts, the AdyenWay is about direct, fast and effective interactions between teams (support, security, development, product). As such, detection and communication happened rapidly and efficiently.
We have a centralized way of understanding services, versions and dependencies. We didn't limit it by looking at which Java application is affected, or which framework can be compromised, but we assessed the overall status of the platform and performed a full review to figure out every potential exploitable behavior and further detection techniques to be implemented.
Although Log4j and several Open Source frameworks depending on it were found in various parts of the Adyen Core platform, no breach or exploitation was recorded thanks to the mitigation factors already in place.
The mitigations were effective and included a mix of best practices and modern security measures:
- Outbound connections (Egress Traffic) are strictly regulated and allowed only when absolutely necessary both on system and application level.
- Systems are kept up-to-date with JDK releases and library patches.
- Good application design and maintenance enables us to patch Log4j versions rapidly without introducing incompatibilities or breaking changes.
- JNDI usage on the platform is strongly limited by design.
- Thread context is exclusively set within local application information thus not being possible under any circumstances to be controlled by user input/crafted requests.
- Adopting a "Zero-Trust" network with strong firewall rules to limit the impact of a breach.
- "Principle of Least Privilege" where applications and containers never run with any kind of elevated access scope.
- Thorough logging and observability mindset where all networking, system, databases, security and user activities are available to perform thorough threat hunting activities.
- Sharp monitoring, built through extensive logging capabilities and enriched with external sources, which allowed new data points to be constantly updated in order to detect and alert of evolving threats, and respond rapidly even in a high stress situation.
Platform-wide patching and upgrade kicked off as soon as the latest Log4j version (v2.15.0 at that time) was made available after a remarkably fast and coordinated effort by the Log4j committers. Within 12 hours, from first becoming aware of the Apache Log4j vulnerability, the entire Adyen platform was upgraded and the fix was verified by Adyen's Security team.
A second iteration was performed when Log4j v2.16.0 became available to complete the vulnerability correction - we are now keeping a close look to monitor other possible implications, corrections and/or upgrades if necessary.
Our merchants and partners are important to us and they are always our first priority: we informed relevant stakeholders about the situation and the activities undertaken to secure the platform and every integration.
"Communication can be informal but effective: there is at least one person coordinating actions, sometimes more depending on the domain or environment. We follow light but clear processes, and involve the right people at the right time."
Analia OrtelladoAdyen SecurityFind below the timeline and main events of the day:
- h08:05 Adyen engineers became aware of the vulnerability.
- h09:15 Apache Log4j vulnerability published by NIST.
- h10:00 Initial Security assessment completed: no impact thanks to mitigating factors in place.
- h11:50 Concluded review of the entire Adyen platform: no instances could be exploited.
- h14:05 Developing and testing new releases complete.
- h17:00 Platform-wide patching and testing (Apache Log4j upgrade).
- h17:45 Security analysis conclusion to confirm health of platform.
Engineering best practices are a form of mitigation in cases like this, so I would like to share tips and practices that are important to consider:
Make sure that you maintain your dependencies to avoid vulnerabilities, deprecation, and etc. Tools never solve all your problems, and dependency management is a complex topic in large codebases, but dependency scanners are easy to deploy in your CI/CD pipelines and there are amazing Open Source options that could help engineering teams to step up their visibility.
Make sure that your software can run in modern and maintained tech stacks. If you are using Log4j version 1 or running in Java SE 6, you will likely have a hard time trying to maintain your application and dependencies.
Sanitize logging statement: this is a must-have OWASP rule.
Reduce usage of dangerous tools, protocols, technology stacks that are not really required but only nice-to-haves. JNDI has been long considered a dangerous interface that needs to be treated with care.
Avoid storing sensitive data (secret keys for example) in environment variables: there is a good possibility data gets exposed by an application crash or some unwanted logging (printenv), and definitely critical if the server is compromised. Integrate strong distributed secret stores instead.
Log only what is strictly necessary: capture user input if it brings value, keeping also in mind that confidential data might end up in the logging streams.
Revise the logging format: a popular pattern layout is `%-5p [%t]: %m%n` which looks harmless until you decide the user can somehow name the Java thread.
A final note to say THANK YOU to the Log4j contributors and all other developers, researchers and communities involved in addressing the security flaw and releasing a new and safer version of one of the most popular Java libraries.
At Adyen, we firmly believe in Open Source (we already sponsor and contribute to projects) and we plan to renew our commitment by open-sourcing more of our great software and supporting projects and committers who dedicate time and passion to those projects.
If you have questions, please contact us on Twitter @AdyenDevs - the team and I are always happy to help.
Stay safe and have a happy end of the year.
October 5, 20215 Minutes
August 26, 20215 Minutes
August 23, 202112 Minutes
By submitting this form, you acknowledge that you have reviewed the terms of our Privacy Statement and consent to the use of data in accordance therewith.
Attachments
- Original Link
- Original Document
- Permalink
Disclaimer
Adyen NV published this content on 22 December 2021 and is solely responsible for the information contained therein. Distributed by Public, unedited and unaltered, on 22 December 2021 13:56:07 UTC.
