VMware : Augmented MISP Integration with NSX Advanced Threat Analyzer

Contributors: Jason Zhang (NSBU TAU), Stefano Ortolani (NSBU TAU)

Introduction

Formerly known as the Malware Information Sharing Platform, MISP is a leading open-source threat intelligence platform (TIP) that organizations of all sizes can leverage to store, share, and enrich threat indicators of compromise (IoCs).

The MISP ecosystem primarily comprises two parts: MISP core (or engine) and MISP modules. MISP core is responsible for the main functionality of the platform, while MISP modules were introduced to extend the capability of MISP without changing MISP core components.

Thanks to the simple API interface provided by MISP, many third-party MISP modules have been developed to greatly extend MISP's capabilities. There are mainly three types of MISP modules: expansion modules, import modules, and export modules. More details on MISP modules can be found on MISP's GitHub MISP module repository, which includes three modules developed by Lastline (now part of VMware) that integrate MISP with VMware NSX Advanced Threat Analyzer (ATA), as we reported earlier.

Recently VMware's Threat Analysis Unit (TAU) developed a new expansion module, which replaces the three Lastline modules. The improvements from the new module are twofold: a simplified enrichment process and an augmented enrichment capability.

In this blog post, we first introduce the newly developed VMware_NSX module, covering its core features, the basic module settings, and the associated IoC enrichment workflow. We then provide two enrichment examples to demonstrate the module's core features and flexibility.

VMware_NSX Module

NSX sandbox is a key component of VMware NSX ATA. The VMware_NSX module is designed to integrate MISP with NSX sandbox so that the results of file detonation from the sandbox, such as detection score, analysis labels, and MITRE ATT&CK techniques can be used to enrich the IoCs associated with malware samples. The VMware_NSX module currently supports file and URL enrichments.

Like other MISP modules, the VMware_NSX module needs to be installed and enabled with the necessary settings configured before it can be invoked to enrich a given IoC. We won't discuss the details of how to install and enable a MISP module in this post, as the process of MISP module installation and enabling is well-documented on MISP's GitHub MISP module repository.

Once the VMware_NSX module is configured correctly, the module can be invoked by clicking the "*" icon (enrichment action symbol) next to a supported IoC. Figure 1 highlights the enrichment symbol with the mouse hover pop-up text "Propose enrichment."

Figure 1: IoC enrichment action symbol.

Upon clicking the enrichment icon, a pop-up window displays (see Figure 2), showing a list of enabled modules that support enrichment for the selected IoC. As we can see, there are two modules in Figure 2: virustotal and vmware_nsx. Both modules can be used to enrich the file hash (SHA1) shown in Figure 1.

Figure 2: Selecting the VMware NSX enrichment module.

Selecting the option highlighted in Figure 2 prompts the VMware_NSX module to start the enrichment process.

To help understand the enrichment process, in the following section we will introduce the basic module settings, such as various API configurations, and explain the role of each API in an enrichment process.

Module Settings

The module settings can be accessed via the MISP web UI interface under server settings & maintenance menu. The enrichment process performed by the VMware_NSX module mainly comprises three API endpoints and related settings, as highlighted in Figure 3.

Figure 3: Basis module settings.

Analysis API

The Analysis API is needed when retrieving an analysis report from NSX ATA for a given IoC (such as a file hash, like MD5, SHA1, or SHA256) or when submitting a file (such as a file attachment or downloaded file) to NSX sandbox for analysis.

A valid pair of key and API token is required to authorize access to the API.

The Analysis_url field is only required for on-premises infrastructures. If it is left empty (which it is by default), the default hosted infrastructure analysis URL settings will be used.

VirusTotal API

VirusTotal API is used to download files from VirusTotal for analysis if a sandbox analysis report is not available at the time of enrichment.

A valid API key is needed to call the API.

MISP API

MISP provides an external API for automating enrichment and other tasks using PyMISP, a Python library to access MISP instances via API. For example, it is common to use PyMISP to enrich a file hash attribute on a MISP instance with the MITRE Attack Pattern, one of MISP's Galaxy clusters.

A MISP instance URL and a valid API key for the instance are required to authorize access to the API. The API key is available in the event actions menu under automation in a MISP instance account.

The APIs discussed above will be invoked at different stages of an enrichment process. Next, we discuss the workflow of IoC enrichment and illustrate how the module handles various scenarios during an enrichment process.

IoC Enrichment Workflow

Figure 4 shows the workflow of IoC enrichment with the VMware_NSX module. When an IoC, such as a file hash or file attachment, is selected for enrichment, the module is invoked, and the enrichment process begins.

Figure 4: The workflow of IoC enrichment via the VMware NSX MISP module.

As seen in Figure 4, the enrichment workflow comprises various steps that are labelled with upper case letters such as A, B, and C. We can break down the workflow into the following processes:

• Analysis report available: A -> B

If a sandbox analysis report is available for a given file IoC, the enrichment process only comprises steps A and B, which is the shortest process among all the scenarios. To carry out the enrichment, the module will invoke the Analysis API to retrieve the most recent analysis report. The report is then used to generate the enrichment results, such as MITRE Attack Pattern IDs and analysis labels, file and network objects, etc.

• Analysis report not available & file download not needed: A->C->D->H->A->B

This process has extra steps C, D, H, and A before reaching step B to obtain enrichment results. As Figure 4 illustrates, the selected IoC is a file attachment but there is no sandbox analysis report for this file, so it needs to be submitted for analysis first. When the sandbox detonation analysis is completed, a report for the file becomes available. Then the enrichment process restarts (either manually or automatically) from step A and ends at step B.

• Analysis report not available & file download needed: A->C->E->F->H->A->B

In this case, the process needs even more steps than the previous scenario before an analysis report can be retrieved (step B). According to Figure 4, this IoC is a file hash but there is no analysis report for this file. To enrich this file hash, the related file needs to be downloaded from VirusTotal first by invoking the VirusTotal API. Assuming the file is successfully downloaded, it is then submitted to NSX sandbox for analysis (step F). The remaining steps H, A, and B are the same as discussed above.

• Analysis report not available & file not available: A->C->E->G

In this scenario, the selected IoC for enrichment is a file hash but there is no analysis report for this file and the file download from VirusTotal is not successful (either the file is not on VirusTotal or the download process fails for other reasons). In such cases, the enrichment process fails and terminates at step G.

IoC Enrichment Examples

As discussed in the workflow section above, the availability of an NSX sandbox analysis report largely affects how an enrichment process develops. In this section, we provide two such examples to demonstrate how the module handles different enrichment scenarios.

Example 1 - NSX sandbox analysis report available

In this example, a file hash will be enriched with an available NSX sandbox analysis report. The enrichment process is the same as:

Analysis report available: A->B (see Figure 4).

The example event only contains one tag "test" and one attribute (a file SHA1), as shown in Figure 5 and Figure 6.

Figure 5: An example MISP event to test IoC enrichment when a sandbox report is available.

Figure 6: The file SHA1 to be enriched with an available VMware NSX sandbox report.

After selecting the VMware_NSX module to start the enrichment process (as described in Figure 2) for the file shown in Figure 6, the suggested enrichment results based on the retrieved analysis report are presented to the user (see Figure 7).

Figure 7: Enrichment results to be added to the MISP event.

The enrichment results include a list of tags, which includes a few MITRE Attack Pattern IDs such as misp-galaxy:mitre-attack-pattern="Browser Bookmark Discovery - T1217″ and misp-galaxy:mitre-attack-pattern="Disabling Security Tools - T1089", as well as some analysis labels from the analysis report like av-cls:ransomware and av-fam:sodinokibi. There are also several objects created as part of the enrichment process.

After clicking the Submit button on the enrichment results page (see Figure 7), the MISP event is then enriched with the suggested tags and various objects, as shown in Figure 8 and Figure 9.

Figure 8: A few Galaxy MITRE ATT&CK IDs, analysis labels and new objects were added to the event after enrichment.

Typical objects created by the enrichment process include a file object, a sandbox report object, and a sandbox signature (or detection) object. As Figure 9 shows, the file object contains various types of file hash and a MIME type for the file. The sandbox-report object shows the sandbox detection score, the report permalink, and other information. The sandbox signature object, named sb-signature in Figure 9, provides the description of detection rules triggered during file analysis.

Figure 9: File metadata, sandbox report, and detection objects.

Example 2 - NSX sandbox analysis report not available

In the next example, the file hash to be enriched is new to NSX sandbox at the time of enrichment, and therefore there is no analysis report for the file. As a result, the file needs first to be downloaded from VirusTotal. This scenario reflects the process shown in Figure 4:

Analysis report not available & file download needed: A->C->E->F->H->A->B.

The test MISP event with a single attribute (a file SHA1) is shown in Figure 10 and Figure 11.

Figure 10: An example MISP event to test IoC enrichment when a sandbox analysis report is not available.

Figure 11: The file SHA1 to be enriched with NSX sandbox analysis.

To enrich the file shown in Figure 11, the VMware_NSX module first invokes the VirusTotal API to download the file. Once downloaded, the enrichment process continues by calling the Analysis API to submit the file for analysis. As shown in Figure 12, two tags were added to the hash after the downloaded file was submitted for analysis: the workflow:state='incomplete' tag implies that the analysis is undergoing, and the vt:download tag means the file is downloaded from VirusTotal.

Figure 12: Workflow state and file download source tags were added.

Once the analysis is completed, the analysis report becomes available and the file IoC can be enriched with a process similar to that discussed in Example 1. As a result, the enrichment process becomes:

Analysis report available: A->B (see Figure 4).

Figure 13 and Figure 14 show the corresponding enrichment results.

Figure 13: A Galaxy MITRE ATT&CK ID, a few analysis labels, and new objects were added to the event after enrichment.

In this example, one MITRE Attack Pattern ID and three analysis labels were tagged to the event (see Figure 13). The enrichment results also include three additional objects, as shown in Figure 14.

Figure 14: File metadata, sandbox report, and detection objects.

Conclusion

In this report, we introduce a new MISP expansion module, VMware_NSX, which is an improved version of the existing MISP modules we previously developed. The new module greatly simplifies the integration of MISP with VMware NSX ATA, and further augments the enrichment process with MISP. As discussed in the IoC enrichment workflow section, the new module can handle various scenarios. For example, as demonstrated above, the module can automatically download a file from VirusTotal and submit it for analysis if a sandbox analysis report is not available for a given file hash.

The VMware_NSX module is available on MISP's GitHub MISP module repository. We plan to deprecate the three Lastline modules from the MISP module repository in December 2021.