Splunk : Approaching Linux Post-Exploitation with Splunk Attack Range

Share:

By Rod Soto January 13, 2022

With the recent release of Sysmon (System Monitor) for Linux by Microsoft, new opportunities for monitoring, detection development, and defense are now possible. Sysmon for Windows is a very popular tool among detection developers and blue teamers as it provides extensive details from system activity and windows logs. Due to the extensive information this service/driver provides in Microsoft Windows, it is very useful when researching attacks and replicating malicious payloads on lab machines.

Enter Splunk Sysmon for Linux Add-On

The new Sysmon for Linux add-on by Cedric HIEN, available for download at Splunkbase (splunkbase.splunk.com) allows us to ingest data and investigate attacks on Linux hosts. Next, we need an easy way to instrument a Linux machine to streamline payload execution and collection of attack artifacts. Of course, we are going to use the Splunk Attack Range to take a look at Linux Sysmon.

As shown, above we can build an Attack Range with Sysmon for Linux and we can then proceed to use that host to replicate attacks. I decided to test some tools that are focused on Linux post-exploitation. I have become familiar with my exploitation research work. I built an Attack Range with Sysmon for Linux enabled and researched what artifacts are created after executing popular Linux post-exploitation tools such as Linpeas, Mimipenguin, linuxexploitsuggester and pspy.

A lot of these tools follow similar commands and some of these tools have used references such as the Linus privilege escalation checklist. Many of these commands seen in an isolated context will not provide a single effective indicator of post-exploitation as they are used for legitimate purposes by administrators. However, thanks to the new Sysmon for Linux we can take advantage of large amounts of previously-unavailable data to delve further into the different processes, services, and user-session information that may indicate the use of these tools in the context of post-exploitation.

Linpeas

Linpeas is a popular tool used to search for possible paths to escalate privileges on Linux/Unix/MacOS hosts. As seen in the following screenshots, the execution of this tool is self-explanatory as it performs a wide variety of checks.

Tool checks for OS version, libraries, and system stats.

Linpeas also enumerates possible exploits available for the target host based on system, service, and library information, and version levels.

Below is another example of privilege escalation checks via SUID executable permissions.

Detection

Exploit suggestion tools in Linux post-exploitation frequently check for the existence and make use of the "strings" and "grep" system utilities. These tools will extensively and recursively check for important system files such as /etc/shadow and, /etc/passwd or directories such as /usr/bin, or /tmp (among others). During these tests, I found that if the system does not have "strings" installed, some of these tools cannot execute.

Sample Event

Below is an initial search that will give us plenty of information on how this tool behaves when executed.

As seen above this tool is very verbose and shows extensive use of the system grep tool.

LinuxExploitSuggester

LinuxExploitSuggester, developed and maintained by Z-labs, is another popular tool used to audit privilege escalation in Linux systems. Essentially, it looks for potentially exploitable local vulnerabilities that may allow operators to escalate privileges. This is a great tool for penetration testers - but as we know all too well, malicious actors often take advantage of defensive security tools to cause harm. As with other tools, LinuxExploitSuggester initially checks for the Kernel and OS version to find related CVEs and Exploits.

As seen above, the tool prints out results based on these initial checks. This enables detection by examining initial checks such as the use of "uname" utility (to check for Linux version and kernel version) and other command checks as shown in the banner. However, there is a particularly strong signature that gives this tool away - the use of grep and the string "exploit-DB". exploit-DB provides the operator with a link to suitable exploits that can be downloaded from exploit-DB.com, a community reference for both penetration testers and malicious hackers.

If found in a production environment, these two processes together would be very unusual unless that environment were under test.

MimiPenguin

MimiPenguin, named after the Windows password extraction tool MimiKatz, attempts to grab passwords from targeted Linux systems. Like the other tools we have discussed, it requires "grep" and "strings" before it can execute and targets specific operating systems and applications such as Kali Linux, GNOME Keyring, LightDM (*nix display managers), Apache2 http-basic authentication, and VSFTD FTP . Its high frequency of "grep", "strings", and "ps" processes are used to fingerprint the operating system, services, and applications. Finally, it attempts to grab passwords from these sources.

This tool did not work well in the attack range as it requires several libraries not present. Even when the precompiled version malfunctions. However, we can still develop signatures by reading the source code. For example, it greps for specific strings such to detect VSFTPD ( ps -eo pid,user,command | grep vsftpd | grep nobody | awk -F ' ' '{ print $1 }' )or Apache2 configurations ( ps -eo pid,user,command | grep apache2 | grep -v 'grep' | awk -F ' ' '{ print $1 }'. ). Although this tool did not work on the range out of the box, it did work on another lab system, successfully extracting passwords as seen above.

Pspy

Pspy is a post-exploitation tool that does not require root to run and allows the operator to snoop processes executed by other users or services. It also allows the operator to enumerate the operating system and detect possible processes suitable for privilege escalation (i.e cron jobs).

Sample Event

Detection

Initially, pspy was difficult to detect on Attack Range. We expected a large number of ps calls to enumerate processes on the machine, but pspy uses its own implementation to scan for processes and even detect short-lived processes. By using inotify_add_watch syscalls on /proc, the tool can monitor command execution by users and services by grabbing and parsing information like command line arguments in /proc/PID/*. Evidence of this behavior can be seen in the line: cmdLine, errCmdLine := readFile(fmt.Sprintf("/proc/%d/cmdline", pid), p.maxCmdLength. Using inotify_add_watch pspy to register for and receive alerts in real time whenever a file or the contents of a directory change, helping it catch short-lived files and processes while they exist. The tool can also scan for files and processes on a configurable interval, such as 1000ms,with the option of targeting specific directories. By default pspy watches ' /usr /tmp /etc /home /var /opt'.

The screenshot above shows printout of the pspy execution. The tool shows a large amount of information about processes and users. pspy excels at providing information on scheduled services or unsuspecting users typing commands.

Were we expecting to see many invocations of the ps binary or other binaries, but instead much of that functionality is built into the pspy tool. By using a different tool like forkstat below, we were able to spot a large number of inotify_add_watch syscalls. Furthermore, by looking at the spy source code, we can see that this binary is watching the /proc directory for events related to the creation of processes by implementing the core functionality of ps as seen in the screenshot below.

In the next screenshot, we can see some syscalls (inotifiy_add_watch). inotify_add_watch allows a process to efficiently monitor changes in a directory without the need to repeatedly scan that directory. The screenshot was obtained via forkstat and allows us to pinpoint pids related to pspy.

Unfortunately, these syscalls are not currently registered via Sysmon for Linux. However, they can be found using other tools such as sysdig. During this research, we found several monitoring agents which are very noisy, frequently executing commands and making syscalls which can complicate the detection of pspy and lead to false positives. This noise can lead to false positives. However unless there is a resident process expected to execute at the same rate and by the same user, the frequency of execution of this process may indicate post-exploitation activity, we are not just able to measure it yet via Sysmon for Linux.

AutoSUID

AutoSUID is a great open-source post-exploitation tool that allows the harvesting of SUID executable files as a path to escalate privileges. SUID (set user id) is permission given to a file that allows execution of the file as its owner. Allowing an unprivileged user to execute a file with elevated privileges, such as root, can be a serious vulnerability under certain circumstances.

As seen in the screenshot above from Attack Range with Sysmon for Linux enabled, this tool executes a specific command looking for SUID executable files (perm notation 2000, 4000, and 6000). Essentially, this is a search for executables that may be used to escalate privileges. An excellent reference for this tool is the binary information found at https://gtfobins.github.io/ . GTFOBins "is a curated list of Unix binaries that can be used to bypass local security restrictions in misconfigured systems.". The above screenshot shows no vulnerable SUID files were found in our Attack Range Linux instance.

Detection

There is a clear, concise detection that can be derived from this tool's execution, process path, and directory. It is extremely uncommon for a user to perform this type of permission check unless auditing or troubleshooting binary execution, so the execution of this command is an event that deserves further analysis.

LinEnum

LinEnum is yet another post-exploitation tool that can be very effective. It does not require sudo or root. This tool performs extensive enumeration and footprinting of the targeted host. The screenshots below show its execution.

An operator can collect a large amount of useful information from this tool, including the processes running as root, which users can sudo without password, or simply the list of admin users (visible in the screenshot above). Just as with other tools, we saw frequent execution of commands like "uname" and "grep" as well as access to /proc/ to obtain information from the host.

The process_current_directory is always important to watch as it may help defenders determine if this is a monitoring tool or a script or binary dropped by an attacker, especially when using monitoring agents that will execute very similar commands constantly. Without consideration for the process_current_directory, there is a good chance that monitoring or legitimate tools may be misidentified as malicious. For example, how often does a legitimate user run a binary that is outside their $PATH?.

The above detections are the first approach to discovering and detecting post-exploitation activities in Linux hosts. We were able to successfully simulate these activities using Splunk Attack Range with Sysmon for Linux. While Splunk add-on is in a new release and is likely to be modified, we can start using it immediately to our advantage. Already, detection of Linux post-exploitation is now more accessible thanks to the release of Microsoft Sysmon for Linux.

Contributors

We would like to thank the following for their contributions to this post.

• Lou Stella
• Eric McGinnis