How to Troubleshoot our Splunk HEC Endpoint

[atoulme]

This blog post is part of an ongoing series on OpenTelemetry.

In this blog post, we will explore the best way to check your connection to the HEC endpoint of your Splunk Cloud or Splunk on-premises deployment.

HEC stands for HTTP Event Collector, and is described at length in our documentation.

HEC is a staple of Splunk - it was first introduced at .conf 2015, with a fun demo using the audience’s phones.

Before we start, you should check where your HEC endpoint is located. If you are on a Splunk Cloud instance, your endpoint may use a different hostname.

The documentation gives you a complete rundown. If you are using a Splunk Cloud Platform instance, the URL typically looks like https://http-inputs-<host>.splunkcloud.com/services/collector/event, where <host> is replaced with your Splunk Cloud instance name. 

Please make sure to follow the documentation! This is a common source of confusion.

To dissipate confusion further, you can check that your HEC endpoint is valid and ready to use with a call to the health endpoint of your HEC server.

Use this command to interact with the server (replacing FOO with your Splunk Cloud instance name):

curl "https://http-inputs-FOO.splunkcloud.com/services/collector/health"

If this is the correct endpoint, you will receive a successful response with the following body:

{"text":"HEC is healthy","code":17}

This validates that you are targeting the correct host.

You still may need to proceed further to check that you can indeed send data to Splunk. To do so, you can create a curl command that will send a HEC event to the server.

The documentation offers such an example:

curl "https://mysplunkserver.example.com:8088/services/collector" \
    -H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \
    -d '{"event": "Hello, world!", "sourcetype": "manual"}'

Note the Authorization header. You would replace its value with the access token you would define in your Splunk environment. Please make sure your token is allowed to send data to the correct indexes.

You can then check that the data was indeed ingested by searching Splunk with index=* sourcetype=manual.

I hope this example was useful! Please feel free to reply in your comments with your own tips and tricks or provide feedback on the documentation. Your suggestions are most welcome!

— Antoine Toulme, Senior Engineering Manager, Blockchain & DLT