Community Blog
Get the latest updates on the Splunk Community, including member experiences, product education, events, and more!

Everything you Wanted to Know About Sending Logs to Splunk (With the new OpenTelemetry Collector)

atoulme
Splunk Employee
Splunk Employee

This blog post is part of an ongoing series on OpenTelemetry.

Curious about OpenTelemetry but more interested in logs than APM tracing or metrics? Look no further! This blog post will walk you through your first OpenTelemetry Logging pipeline...

WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support the use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud.

The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF). The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, and Helm among others.

OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes, and exports data as a pipeline.

The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. Please see our official documentation to install the collector.

At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples.

The OpenTelemetry collector works using a configuration file written in YAML.

The collector defines metrics pipelines. For our case, we have defined a pipeline that reads from a file and sends its data to Splunk.

 

 

 

   pipelines:
      logs:
        receivers: [filelog]
        processors: [batch]
        exporters: [splunk_hec/logs]

 

 

 

We use a processor named the batch processor to place multiple entries in one payload, so we avoid sending many messages at once to Splunk.

Note we have placed our pipeline under the name logs. That means we intend to use this pipeline to ingest log records. If you have multiple log pipelines, they must start with logs, followed by a slash, and a unique name, such as:

 

 

 

   pipelines:
      logs:
        …
      logs/other:
        …
      logs/anotherone:
        …

 

 

 

This notation is also used for other components, such as filelog or splunk_hec in our example.

Going back to our pipeline, we have defined it to contain a receiver named "filelog". We have not written it down yet, so let’s write down our receivers:

 

 

 

receivers:
    filelog:
      include: [ /output/file.log ]

 

 

 

Our filelog receiver will follow the file /output/file.log and ingest its contents.

We also need to define our Splunk HEC exporter. Here is our exporters section:

 

 

 

exporters:
    splunk_hec/logs:
        token: "00000000-0000-0000-0000-0000000000000"
        endpoint: "https://splunk:8088/services/collector"
        source: "output"
        index: "logs"
        max_connections: 20
        disable_compression: false
        timeout: 10s
        tls:
          insecure_skip_verify: true

 

 

 

This exporter defines the configuration settings of a Splunk HEC endpoint. More documentation and examples are available as part of the OpenTelemetry Collector Contrib github repository.

This particular Splunk endpoint says it will send data to the logs index, under the source “output”, to a Splunk instance located under the Splunk hostname, with a HEC token that is just a set of zeroes.

We’re now going to set all the pieces in motion to deliver to you this example end to end.

First, we are going to define a program that outputs data to a file.

 

 

 

bash -c "while(true) do echo \"$$(date) new message\" >> /output/file.log ; sleep 1; done"

 

 

 

This bash script will send the current date, accompanied with "new message", every second, until told to stop.

Second, we prepare a simple Splunk Enterprise Docker container to run for this example.

We set up its logs index with a splunk.yml configuration file:

 

 

 

splunk:
    conf:
      indexes:
        directory: /opt/splunk/etc/apps/search/local
        content:
          logs:
            coldPath: $SPLUNK_DB/logs/colddb
            datatype: event
            homePath: $SPLUNK_DB/logs/db
            maxTotalDataSizeMB: 512000
            thawedPath: $SPLUNK_DB/logs/thaweddb

 

 

 

We load up this file by mounting as a volume. We also run the container to set up a default HEC token, open ports, accept the Splunk license, and set a default admin password. Obviously, this is only useful here for our demonstration. There are more interesting configuration possibilities if you follow along this Github repository for Splunk Docker, and be sure to check out Splunk Operator for larger, production-grade deployments.

All told, our Splunk server looks like this in our Docker Compose:

 

 

 

# Splunk Enterprise server:
  splunk:
    image: splunk/splunk:latest
    container_name: splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_HEC_TOKEN=00000000-0000-0000-0000-0000000000000
      - SPLUNK_PASSWORD=changeme
    ports:
      - 18000:8000
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:8000']
      interval: 5s
      timeout: 5s
      retries: 20
    volumes:
      - ./splunk.yml:/tmp/defaults/default.yml
      - /opt/splunk/var
      - /opt/splunk/etc

 

 

 

With that, you are ready to try out our example.

To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed.

First, check out the repository using git clone:

 

 

 

git clone https://github.com/signalfx/splunk-otel-collector.git

 

 

 

Using a terminal window, navigate to the folder examples/otel-logs-splunk.

Type:

 

 

 

docker-compose up

 

 

 

This will start the OpenTelemetry Collector, our bash script generating data, and Splunk Enterprise.

Your terminal will display information as Splunk starts. Eventually, Splunk will display this table to let you know it is available:

 

 

 

splunk           | Wednesday 04 May 2022  02:04:18 +0000 (0:00:00.818)       0:00:47.544 ********* 

splunk           | =============================================================================== 

splunk           | splunk_common : Start Splunk via CLI ----------------------------------- 11.97s

splunk           | splunk_common : Update Splunk directory owner --------------------------- 6.45s

splunk           | splunk_common : Update /opt/splunk/etc ---------------------------------- 5.42s

splunk           | splunk_common : Get Splunk status --------------------------------------- 2.50s

splunk           | Gathering Facts --------------------------------------------------------- 1.67s

splunk           | splunk_common : Hash the password --------------------------------------- 1.23s

splunk           | splunk_common : Set options in logs ------------------------------------- 1.11s

splunk           | splunk_common : Test basic https endpoint ------------------------------- 1.00s

splunk           | Check for required restarts --------------------------------------------- 0.82s

splunk           | splunk_standalone : Check for required restarts ------------------------- 0.77s

splunk           | splunk_standalone : Update HEC token configuration ---------------------- 0.72s

splunk           | splunk_standalone : Get existing HEC token ------------------------------ 0.69s

splunk           | splunk_standalone : Setup global HEC ------------------------------------ 0.69s

splunk           | splunk_common : Check for scloud ---------------------------------------- 0.66s

splunk           | splunk_common : Generate user-seed.conf (Linux) ------------------------- 0.63s

splunk           | splunk_common : Find manifests ------------------------------------------ 0.54s

splunk           | splunk_common : Wait for splunkd management port ------------------------ 0.43s

splunk           | splunk_common : Cleanup Splunk runtime files ---------------------------- 0.40s

splunk           | splunk_common : Check for existing splunk secret ------------------------ 0.30s

splunk           | splunk_common : Check for existing installation ------------------------- 0.30s

splunk           | ===============================================================================

splunk           | 

splunk           | Ansible playbook complete, will begin streaming splunkd_stderr.log

 

 

 

Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme.

You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application.

In this application, enter this search to look for logs:

 

 

 

index="logs"

 

 

 

The latest logs generated by the bash script will show:

atoulme_0-1660609696127.png

After exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! With this example, you have deployed a simple pipeline to ingest the contents of a file into Splunk Enterprise.

If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository.

— Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...