This blog post is part of an ongoing series on OpenTelemetry.
Curious about OpenTelemetry but more interested in logs than APM tracing or metrics? Look no further! This blog post will walk you through your first OpenTelemetry Logging pipeline...
WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION. When sending data to Splunk Enterprise, we currently only support the use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud.
The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF). The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, and Helm among others.
OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes, and exports data as a pipeline.
The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. Please see our official documentation to install the collector.
We use a processor named the batch processor to place multiple entries in one payload, so we avoid sending many messages at once to Splunk.
Note we have placed our pipeline under the name logs. That means we intend to use this pipeline to ingest log records. If you have multiple log pipelines, they must start with logs, followed by a slash, and a unique name, such as:
This particular Splunk endpoint says it will send data to the logs index, under the source “output”, to a Splunk instance located under the Splunk hostname, with a HEC token that is just a set of zeroes.
We’re now going to set all the pieces in motion to deliver to you this example end to end.
First, we are going to define a program that outputs data to a file.
bash -c "while(true) do echo \"$$(date) new message\" >> /output/file.log ; sleep 1; done"
This bash script will send the current date, accompanied with "new message", every second, until told to stop.
Second, we prepare a simple Splunk Enterprise Docker container to run for this example.
We set up its logs index with a splunk.yml configuration file:
We load up this file by mounting as a volume. We also run the container to set up a default HEC token, open ports, accept the Splunk license, and set a default admin password. Obviously, this is only useful here for our demonstration. There are more interesting configuration possibilities if you follow along this Github repository for Splunk Docker, and be sure to check out Splunk Operator for larger, production-grade deployments.
All told, our Splunk server looks like this in our Docker Compose:
You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application.
In this application, enter this search to look for logs:
The latest logs generated by the bash script will show:
After exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! With this example, you have deployed a simple pipeline to ingest the contents of a file into Splunk Enterprise.