This blog post is part of an ongoing series on OpenTelemetry.
The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF) The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, or Helm among others.
OpenTelemetry defines a model to represent traces, metrics and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support.
At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples. This blog post will walk you through sending metrics data with the OpenTelemetry Collector to a Splunk Enterprise deployment.
WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION. When sending data to Splunk Enterprise, we currently only support use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud.
At our dinner table, I am often asked this question: “How can I send Prometheus metrics data to Splunk Enterprise?” Sometimes, I have to remind my kids to ask me. I also field Harry Potter and Minecraft topics, albeit without as much enthusiasm.
In this blog post, we will set up a simple program emitting Prometheus metrics, and we will use the new OpenTelemetry Collector to read those metrics and send them to Splunk. Before we start, I should mention that Splunk Enterprise has a specialized type of index for metrics. If you came here from our previous post on ingesting log data, you will notice that this time we add a bit more information when we define our index in splunk.yml:
splunk:
conf:
indexes:
directory: /opt/splunk/etc/apps/search/local
content:
metrics:
coldPath: $SPLUNK_DB/metrics/colddb
datatype: metric
homePath: $SPLUNK_DB/metrics/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/metrics/thaweddb
Notice the datatype: metric addition. If your index is not declared as a metric index, you will not be able to ingest metrics through Splunk HEC. To learn more about our metric index technology, please start with our Splunk docs website.
Now that we have gotten this out of the way, let’s make sure we have Prometheus metrics to show. Prometheus metrics are exposed on a http endpoint that can be scraped (collected, in OpenTelemetry parlance). We are going to expose a static file that contains valid Prometheus data over http to expose those metrics to our collector.
In a Docker compose environment, we would define a container that serves http data.
httpd:
image: dragas/thttpd
container_name: thttpd
restart: always
volumes:
- ./content:/var/www/http:ro
ports:
- 80:80
This container is pointing at a content folder. In the folder, we place a file named “metrics”:
# HELP foo_bar The amount of foo going well with bar
# TYPE foo_bar gauge
foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 3732552.0
foo_bar{application="my-own-app",area="heap",id="G1 Eden Space",} 7.9641776E7
foo_bar{application="my-own-app",area="nonheap",id="Metaspace",} 3.72735E7
foo_bar{application="my-own-app",area="heap",id="G1 Survivor Space",} 2097152.0
foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'non-nmethods'",} 2544904.0
foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.0555296E7
foo_bar{application="my-own-app",area="nonheap",id="Compressed Class Space",} 4325676.0
foo_bar{application="my-own-app",area="heap",id="G1 Old Gen",} 5.0331678E7
This file contains JVM metrics related to garbage collection.
Now, we’re going to create a metrics pipeline in the collector. This is similar to our last example, with a new receiver named “prometheus”:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [splunk_hec/metrics]
The prometheus receiver has a configuration that mirrors the usual Prometheus scrape configuration:
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['thttpd:80']
Finally, we set up our Splunk exporter:
exporters:
splunk_hec/metrics:
# Splunk HTTP Event Collector token.
token: "00000000-0000-0000-0000-0000000000000"
# URL to a Splunk instance to send data to.
endpoint: "https://splunk:8088/services/collector"
# Optional Splunk source: https://docs.splunk.com/Splexicon:Source
source: "app"
# Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
sourcetype: "jvm_metrics"
# Splunk index, optional name of the Splunk index targeted.
index: "metrics"
# Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
max_connections: 20
# Whether to disable gzip compression over HTTP. Defaults to false.
disable_compression: false
# HTTP timeout when sending data. Defaults to 10s.
timeout: 10s
# Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
# For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
insecure_skip_verify: true
Note that we choose the name splunk_hec/metrics, and we pick the metrics index, but our configuration is the same as the way we set ourselves up to ingest logs otherwise.
We have put this all together into an example that lives under Splunk’s OpenTelemetry Collector github repository. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed.
First, check out the repository using git clone:
git clone https://github.com/signalfx/splunk-otel-collector.git
Using a terminal window, navigate to the folder examples/splunk-hec-prometheus.
Type:
docker-compose up
This will start the OpenTelemetry Collector, our bash script generating data, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display the same information as in our last blog post to let us know it is ready.
Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme.
You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. In this application, click the Analytics Workspace to see our metrics data.
Click on the foo_bar metric. You will see a line of data points shown in the graph.
In the right-hand panel, click on Split By, and select id. You will now see your data split by its id metric field, as below:
You can perform further calculations, such as bucketing values by data or rendering it in a dashboard.
When you have finished exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! This concludes our Prometheus metrics example. We have used the OpenTelemetry Collector to successfully scrape Prometheus metrics and send them to our Splunk Enterprise instance.
If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository.
— Antoine Toulme, Senior Engineering Manager, Blockchain & DLT
