Splunk : Stream Your AWS Services Metrics to Splunk

Share:

By Anush Jayaraman May 07, 2021

Amazon Web Services (AWS) recently announced the launch of CloudWatch Metric Streams. Cloudwatch Streams can stream metrics from a number of different AWS resources using Amazon Kinesis Data Firehose to target destinations. The new service is different from the current architecture. Instead of polling, metrics are delivered via an Amazon Kinesis Data Firehose stream. This is a highly scalable and far more efficient way to retrieve AWS service metrics.

What this means for current Splunk customers is they now have the option of either using the Splunk Add-on of AWS to poll metrics or to make use of this new service and let Amazon Kinesis Data Firehose push metrics to a Splunk HEC endpoint, and reduce their latency by anywhere between 5 to 10 minutes.

CloudWatch Metric Streams are fully managed and very easy to set up. Streams can scale to handle any volume of metrics, with delivery to the destination within two or three minutes. You can choose to send all available metrics to each stream that you create, or you can opt-in to any of the available AWS (EC2, S3, and so forth) or custom namespaces.

The CloudWatch Metrics Streams are available under the CloudWatch service. The configuration is quick and easy. Simply create a stream by selecting the namespaces or services for which metrics are available and by selecting a Firehose resource that is configured to send data to a Splunk HEC endpoint. Remember to set the HEC endpoint type as an event. More information on configuring Amazon Kinesis Data Firehose to send data to Splunk is available here.

Once a stream has been set up, metrics start to flow within a minute or two. The flow can be stopped and restarted later if necessary, which can be handy for testing and debugging. When you set up a stream, you choose between the binary Open Telemetry 0.7 format and the human-readable JSON format.

Though the data is in JSON format the existing source types in the Splunk AWS add-on for interpreting Cloudwatch metrics will not be able to decode the data properly, as the data structure with streams is a bit different. To let our customers make use of existing metric dashboards and searches from the Splunk App for AWS, Splunk is also providing a serverless function to transform the new metrics data into Splunk's Cloudwatch source type-specific format.

The lambda transformation can be attached to the Kinesis Data Firehose that will be delivering Cloudwatch streams metric data to Splunk HEC. As of now, the transformation function supports conversion of the JSON output only. The OTEL output format of metrics is simply sent to Splunk without any normalization or enrichment.

The serverless function splunk-aws-cloudwatch-streaming-metrics-processor is now available on AWS Serverless Application Repository for use. The source code for the function is also available here.

Splunk also supports streaming CloudWatch metrics to our Splunk Infrastructure Monitoring solution. More details on that are available in our blog post, Low Latency Observability Into AWS Services With Splunk.