Extreme : SolarWinds – A Supply Chain Compromise

Early in the year, 2020 life changed for a lot of us. Many would think of the COVID virus and that certainly is true. But I am referring to something different. Something that is just as scary. I am referring to Sunburst, which is a compromise of the SolarWinds network and systems management platform. Unless you have been living under a rock you have heard about this but may not have a good scope on what the attack was and how it operated. What you will see is that this is the most serious supply chain compromise in the history of the industry and that we should look at it as a harbinger of what is to come. No one is immune. If you create and build code, then you need to realize that the same could occur to you. The industry needs a new way of thinking to have any hope of protecting against this or really an understanding as to how weak or vulnerable the supply chain is. The world has changed around us, and we need to realize the change in the level of the playing field that has resulted. If I sound serious, it is because I am. We need to all take note of this event.

Why SolarWinds?

SolarWinds was the ideal target for several reasons. First its purpose is for network and systems management. SolarWinds can talk to almost anything, out of the box. It often will do so with privileged credentials that have purposely installed for the purposes of systems monitoring or configuration change. In essence, any attacker that compromises the network management system will have the same degree of control and reach as the platform itself. This is what is often referred to as 'God mode'.

Another aspect is that as an NMS, SolarWinds talked to everything. Switches, routers, servers, and it often did so again with privileged credentials. In essence, it would be hard to pick out nefarious activity in all of the normal white noise. This was indeed the case as its 9 months of residency in many networks proves out. Due to the nature of the attack, the malware came as a part of normal software updates or installation. Therefore, there is no need for the attacker to establish persistence as it is already obtained if desired. More on that later.

The Compromise

While it is still not certain exactly how the compromise of the supply chain occurred, it is most likely that this was done prior to the build process. It is also evident that SolarWinds 'Team City' build servers were likely the point of compromise. There was a piece of code known as 'Sunspot' which performed the injection of the Sunburst code by utilizing taskhostsvc.exe. Which is a normal Windows internal process for scheduled tasks. The particular file that was modified with the Sunburst backdoor was InventoryManager.cs which is found in the Orion Improvement Business Layer. There was a nifty swap of the file where the compromised code was injected into the build on bootup but would swap back to the original file in the build directory. This created a very stealthy presence on the platform that could only be picked out by analyzing the running source code. Looking at the build directory would show everything as normal. This indicates quite a degree in the sophistication of the attacker, which is believed to be a set of nation-state actors.

Let the stages unfold…

The malware was also quite sophisticated in that it was a multi-stage attack. It also is able to sense the environment to determine if it is in production mode or a sandbox lab environment. Additionally, if it did not sense Internet access the code would typically not activate. Once the environment is determined to be right the Sunburst code would begin to beacon out to a well-aged valid domain that looks quite innocuous. In the beacon would be information about the system and relevant domain information. This was performed in a process that would run on a regular reliable basis, the process is refreshInternal(). But more importantly, the code would wait for a random time offset to activate. In some instances, this could be a couple of months.

Most platforms were not further affected. This was most probably due to a lack of interest in the target system. If you were 'lucky' enough to be selected, the next step would be to establish a command and control (C2). Once again, a long series of innocuous-sounding well-aged domains that could be selected randomly via a custom Domain Generation Algorithm (DGA). This indicates not only sophistication but a long period of prior planning for the compromise to be set up and orchestrated. Once the C2 channel was set up there were a series of roughly 20 'jobs' that the Sunburst malware could execute, ranging from direct commands, to write file/delete file, to task management and systems reboot. There are much more capabilities included as well. Basically, the job set provides complete access and capability to the SolarWinds system platform. From there the platform can provide a very convenient pivot from where to launch further compromises and lateral movement. To reflect this is the SolarLeaks web site, which provides the ability to purchase code bundles from some very large vendors in the industry. Since that time 23 other technology companies have announced compromise and it is suspected that many government agencies have been compromised as well.

Another piece to the puzzle is a piece of malware code referred to as Teardrop. Basically, this was a post exploit that replaced or tampered with system .dll's within NetSetupSvc as well as a memory only dropper. This provided an effective 'trail brush' to any activities of the Sunburst compromise. All in all a very sophisticated compromise, not only of the supply chain but of the SolarWinds platform in question.

An Intentional Kill Switch?

But there is one curious aspect about this compromise. For all of the sophistication of the multiple stages and code actions, the initial beacon was set to a static domain, avsvmcloud.com. This domain has since been sink holed by Microsoft. This is almost amateurish in its method and technique. Due to this, it is suspected that this was an intentional 'kill switch', meaning that the compromise was for a specific purpose and was not intended for 'perpetual' residence. It should be noted that while the sink holing of avsvmcloud.com provided a kill switch for the beaconing, it does not address the secondary or tertiary stages of the attack, namely command and control (C2) and post-exploitation. To address this Fire Eye has posted a series of IOC's (Indicators of Compromise) that can be used with Security Event Information Managers. If you do not have a resident SIEM, the Infosec community has posted a very good video to illustrate how these IOC's can be used in tandem with Snort and Wireshark to detect C2 activities.

Catching C2 Traffic

For several years now, actually, for a decade, I have been making 'Stealth Networking' design recommendations. As a part of this, all network management and administrative interfaces should be isolated as much as possible. Any remaining 'required' traffic is then monitored in a very strict fashion. This approach creates a 'C2 trap'. In a general sense, any anomaly of communications should be investigated as soon as possible. But when combined with the IOC's posted by Fire Eye, an effective wrapper on this compromise can be obtained. Again, note that this does not address any lateral movement that may have been induced into other systems. As a result, strict monitoring of any outbound access to suspected domains should be quantified at the outbound security demarcation. As a general rule you should alert on any new domains that are seen in the logs.

I would like to thank my friends at Fire Eye for the heads up and information details on this compromise. I think that you will agree that this shifts the landscape that we are standing upon. Stay alert, there will be more to come.