Splunk : Getting Started with the Splunk Distribution of OpenTelemetry Python

Share:

By Courtney Gannon April 16, 2021

In our last blog, we introduced OpenTelemetry Python v1.0.0 and walked you through instrumenting a Python application and install both the OpenTelemetry API and SDK. In this blog, we'll show you how to get started with the Splunk Distribution of OpenTelemetry Python- a secure, production-ready, Splunk-supported distribution of the OpenTelemetry project that provides multiple installable packages that automatically instruments your Python application to capture and report distributed traces to Splunk APM (no code changes required!), making it easy to get started with distributed tracing!

This Splunk distribution comes with the following defaults:

• B3 context propagation.
• Jaeger thrift exporter configured to send spans to a locally running SignalFx Smart Agent (http://localhost:9080/v1/trace).
• Unlimited default limits for configuration options to support full-fidelity traces.
  

If you're currently using the SignalFx Tracing Library for Python please note that it has been depreciated, so please migrate to the Splunk Distribution of OpenTelemetry Python (see Migrate from the SignalFx Tracing Library for Python).

The instrumentation works with Python version 3.6+. Supported libraries are listed here.

This project is currently in BETA.

Requirements

If you're still using Python 2, continue using the SignalFx Tracing Library for Python.

Get Started

First, install the splunk-opentelemetry package, run the bootstrap script and wrap your run command with splk-py-trace.

For example, if the runtime parameters were:

python main.py --port=8000

Then the runtime parameters should be updated to:

$ pip install splunk-opentelemetry

$ splk-py-trace-bootstrap

$ SPLUNK_SERVICE_NAME=my-python-app 

    splk-py-trace python main.py --port=8000

Notes:

• Depending on the default python version on your system, you might want to use pip3 and python3 instead.
• To be able to run splk-py-trace and splk-py-trace-bootstrap, the directory pip installs scripts to will have to be on your system's PATH environment variable. Generally, this works out of the box when using virtual environments, installing packages system-wide or in container images. In some cases, pip may install packages into your user local environment. In that case you'll need to add your Python user base's bin directory to your system path. You can find out your Python user base as follows by running python -m site --user-base.
• For example, if python -m site --user-base reports that /Users/splunk/.local as the Python user base, then you can add the directory to your path on Unix like system as follows:
  export PATH='/Users/splunk/.local/bin:$PATH'
  

The service name is the only configuration option that typically needs to be specified. A couple of other configuration options that may need to be changed or set are:

• Endpoint if not sending to a locally running Smart Agent with default configuration
• Environment attribute (example: OTEL_RESOURCE_ATTRIBUTES=environment=production) to specify what environment the span originated from.
  

Instrumentation works by patching supported libraries at runtime with the OpenTelemetry API to capture and export trace spans. The library is 100% interoperable with OpenTelemetry Python.

To see the Python instrumentation in action with sample applications, see our examples.

All Configuration Options:

Environment Variable	Config Option	Default Value	Notes
OTEL_EXPORTER_JAEGER_ENDPOINT	endpoint	http://localhost:9080/v1/trace	The jaeger endpoint to connect to. Currently, only HTTP is supported.
SPLUNK_SERVICE_NAME	service_name	unnamed-python-service	The service name of this Python application.
SPLUNK_ACCESS_TOKEN	access_token		The optional organization access token for trace submission requests.
SPLUNK_MAX_ATTR_LENGTH	max_attr_length	1200	Maximum length of string attribute value in characters. Longer values are truncated.
OTEL_RESOURCE_ATTRIBUTES		unset	Comma-separated list of resource attributes added to every reported span. Example: key1=val1,key2=val2
OTEL_TRACE_ENABLED		true	Globally enables tracer creation and auto- instrumentation.

Exporting to Smart Agent, Otel Collector or SignalFx Ingest

This package exports spans in Jaeger Thrift format over HTTP and supports exporting to the SignalFx Smart Agent, OpenTelemetry collector and directly to SignalFx ingest API. You can use OTEL_EXPORTER_JAEGER_ENDPOINT environment variable to specify an export endpoint. The value must be a full URL including scheme and path.

Smart Agent
This is the default option. You do not need to set any config options if you want to export to the Smart Agent and you are running the agent on the default port (9080). The exporter will default to http://localhost:9080/v1/trace when the environment variable is not specified.

OpenTelemetry Collector

In order to do this, you'll need to enable Jaeger Thrift HTTP receiver on OpenTelemetry Collector and set OTEL_EXPORTER_JAEGER_ENDPOINT to http://localhost:14268/api/traces assuming the collector is reachable via localhost.

SignalFx Ingest API

In order to send traces directly to SignalFx ingest API, you need to:

1. Set OTEL_EXPORTER_JAEGER_ENDPOINT to https://ingest..signalfx.com/v2/trace where realm is your SignalFx realm e.g, https://ingest.us0.signalfx.com/v2/trace.
2. Set SPLUNK_ACCESS_TOKEN to one of your SignalFx APM access tokens.
   

Advanced Getting Started

For advanced and alternative instructions to get started please see the README here.

Voila! You're up and running! You can find more information and FAQs on the docs site.

What's Next?

Next in our blog series, we'll be sharing how to get started with the Splunk Distribution of OpenTelemetry Java and the Splunk Distribution of OpenTelemetry Go. Next up for the project, we'll be announcing OpenTelemetry Java Script v1.0.0 GA soon. Check back here or the OpenTelemetry blog for updates.

ICYMI

• Getting Started with OpenTelemetry Python
• Getting started with OpenTelemetry .NET and OpenTelemetry Java
  

Splunk is a contributor to OpenTelemetry and committed to accelerating the adoption of the project. For more information about Splunk and OpenTelemetry, you can check out our latest OpenTelemetry blog posts.