Trend Micro : Stronger Cloud Security in Azure Functions Using Custom Cloud Container

We have written extensively on the security gaps in Azure Functions and Azure App Services, including their consequences. One way developers can enhance cloud security and minimize these gaps is to create custom container image and use the Distroless approach. In this entry, we veer the conversation toward what skilled developers can do to minimize the impact of these security gaps, specifically in Azure Function.

Azure Functions

Azure Functions is a serverless solution aimed at simplifying the deployment and maintenance of applications for developers.

At its surface, we have the App Service plan, which guarantees physical hardware allocation and which we could imagine as a virtual machine. Inside that, we can find a Docker container engine installed. This engine executes a container image that is built with Azure-function-host runtime. Azure-function-host, by its name, effectively manages the Azure Function Runtime, making it responsible for communication with Azure back ends.

This architecture executes azure-functions-worker when serverless function execution is triggered, which in turn executes the actual serverless application with the provided function code.

Figure 1. Simplified architecture of Azure Functions

Creating a custom container in Azure Function

The default container image for chosen stack could be replaced by a custom container image. In such a case the image must contain the azure-function-host so it can work properly with Azure Functions. It's worth mentioning that the option to create a custom container is only available for the Linux platform on Azure Functions Premium plan.

Figure 2. Creating Azure Function with custom Docker container

For this blog entry, we followed Azure documentation for creating a custom container using Python as our code interpreter. However, we made a slight modification where we chose private container registry inside Azure for the deployment.

Figure 3. Setting private registry for Azure custom container serverless deployment

We built the container image locally, then pushed it into private registry that we configured to be linked with serverless function.

Figure 4. The deployment diagram

Building the image

For our base image, we chose mcr.microsoft.com/azure-functions/python:4-python3.9 from the Azure Functions Base list available inside Microsoft Container Registry.

Now we go back to our aim for this blog entry, which is to better secure the use of Azure Functions without affecting its functionality. This aim can be broken down into three goals:

1. To remove sensitive environmental variables inside the serverless application executing context
2. To reduce the container image and limit permissions needed for our application
3. To minimize the impact of our changes on the functionality of Azure Functions

It's important to note that some environmental variables will likely be required for function-host to run and thus for the whole serverless application to work. However, we want to ensure that our serverless application does not have access to such sensitive variables.

Before we start, we need to identify the differences in the Python stack chosen when creating Azure function based on azure-functions/mesh:3.7.1-python3.9 and when creating the same function using the Azure Function Base-Python image.

Figure 5. Comparison of container images

As illustrated in Figure 5, the mesh container image executes initialization wrapper script under the root user before executing the WebHost.dll binary under the app user using sudo command, thereby passing all the environmental variables to WebHost.dll. In comparison, base images execute WebHost.dll binary under root user by default. The WebHost.dll then executes the python-worker, the process that will then execute the serverless code itself.

Removing sensitive environmental variables

Sensitive environmental variables are needed inside the WebHost.dll for it to run. Because of this nature, sensitive information is inherited into the python-worker process and the serverless code executed out of it. Since the variables are part of process memory, our options for removing them are limited. In addition, we can print other process environmental variables running under the same user by using read permissions and the nature of /proc/ file system.

Figure 6. Accessing other process environmental variables

Because of this feature, the best option is to alter the functionality of WebHost.dll binary (or its configuration) to execute the language-worker under a different user and without the sensitive environmental variables.

Since we already have the container image build process in our hands, we can investigate what is the best alteration point. Since our interpreter is Python, the easiest way to inject our code is to rename the Python binary inside the container image and replace it with a custom shell script under the original name.

Figure 7. Environmental analysis of the container

The content of our shell script will be simple. We execute the Python worker as a different user using the sudo -u www-data command without passing environmental variables.

If a developer would want to pass environmental variables, they can limit access to sensitive variables using unset command and the E parameter of sudo.

Figure 8. Executing the python under user www-data passing all other parameters

As Figure 9 shows, we were able to get rid of environmental variables and limit access to sensitive ones when needed.

Figure 9. Running python worker without additional environmental variables

Figure 10. Denying access to sensitive environmental variables

We also tested whether the changes we made still allowed us to run our serverless function within Azure environment successfully. Figure 11 shows the result of this test.

Figure 11. Custom container running at Azure without environmental variables

The distroless approach: Reducing the container image and limiting permissions

Our second goal was to reduce the container binaries and image size to their bare minimum (the application and its dependencies), a method which is better known as the distroless approach. Using this approach, we will reduce our custom container by removing binaries that are not essential for running the application and could provide useful tools for attackers in the event of a successful exploit.

The binaries we removed from the container image are all binaries from the /bin directory, which includes the shell as well. We would therefore need to update our environmental tweak later. We also removed curl, wget, and perl binaries located in the /usr/bin directory in our demonstration example.

Figure 12. Example of a distroless container Dockerfile

Minimizing the impact of our changes

We now need to minimize the effects of our changes and ensure functionality. Because we had removed the shell interpreter, our script wouldn't have worked, so instead we replaced the script with a custom compiled binary that does the same job. Instead of using shell interpreter, however, we used the execve system function. This function lets us set environmental variables for the new process, allowing us to specify the non-sensitive environmental variables we need in our application, which we can obtain dynamically using getenv function.

Figure 13. Custom binary PoC

Figure 14. Successfully deployed custom image inside Azure Serverless environment, specifying environmental variables available to the executed serverless code

Conclusion

In previous blog entries, we discussed the architectural design flaws we saw in the cloud, which could allow malicious actors to abuse environmental variables upon successful exploitation.

In our entry on the Azure App Services threat model, we showed gaps in the architectural design, such as the use of master root password for the container and environmental variables that contained sensitive information. We explained why it is a bad idea to store sensitive information inside environmental variables, even if the DevOps community might think otherwise. We also described the consequences of keeping sensitive information inside environmental variables.

As mentioned, we are shifting our discussion toward what developers can do to minimize the impact of security gaps in the cloud. We aim to do this by introducing little tweaks to the container image that developers are allowed to produce. Developers should know not only what runs beneath the surface but also that trusting default images has its limits. They should evaluate services carefully and learn to remain vigilant even when using trustworthy services.

Hardening security and maintaining application functionality can be difficult. We proved that it is possible to get rid of environmental variables and transfer non-sensitive environmental variable to the low privileged language worker through proper container image design, so it is not far-fetched to see such security measures performed by platform developers as well.

Tags

Cloud|Articles, News, Reports|Research