All Apps and Add-ons

Kinesis Firehose - Could not connect to the HEC endpoint

Kieffer87
Communicator

We are trying to send data to Splunk HEC via Kinesis Firehose but for some reason Firehose keeps logging "Could not connect to the HEC endpoint. Make sure that the HEC endpoint URL is valid and reachable from Kinesis Firehose." We've tried a combination of the following with no luck:

https://hostname.test.com:8088
https://hostname.test.com:8088/services/collector
https://hostname.test.com:8088/services/collector/raw

We are referencing this post: Power Data Ingestion into Splunk which indicates the first https://hostname.test.com:8088 with a raw endpoint should have worked. I'm able to post events via curl using batch and the raw endpoint and json and the event endpoint. This tells me the ELB is working and forwarding events. So I'm wondering what others have set for their Splunk Cluster Endpoint and Splunk endpoint type in Firehose?

Raw Endpoint:

curl -k "https://hostname.test.com:8088/services/collector?channel=00872DC6-AC83-4EDE-8AFE-8413C3825C4C" -H "Authorization: Splunk token" -d '127.0.0.1 - admin [28/Sep/2016:09:05:26.875 -0700] "GET /servicesNS/admin/launcher/data/ui/views?count=-1 HTTP/1.0" 200 126721 - - - 6ms 127.0.0.1 - admin [28/Sep/2016:09:05:26.917 -0700] "GET /servicesNS/admin/launcher/data/ui/nav/default HTTP/1.0" 200 4367 - - - 6ms 127.0.0.1 - admin [28/Sep/2016:09:05:26.941 -0700] "GET /services/apps/local?search=disabled%3Dfalse&count=-1 HTTP/1.0" 200 31930 - - - 4ms
127.0.0.1 - admin [28/Sep/2016:09:05:26.954 -0700] "GET /services/apps/local?search=disabled%3Dfalse&count=-1 HTTP/1.0" 200 31930 - - - 3ms
127.0.0.1 - admin [28/Sep/2016:09:05:26.968 -0700] "GET /servicesNS/admin/launcher/data/ui/views?digest=1&count=-1 HTTP/1.0" 200 58672 - - - 5ms'

Events Endpoint:

curl -k "https://hostname.test.com:8088/services/collector?channel=00872DC6-AC83-4EDE-8AFE-8413C3825C4C" -H 'Authorization: Splunk token' -d '{"event": "Hello"}'
Tags (1)
0 Karma

ledion
Path Finder

You could use Cribl to pull the data directly from a Kinesis Stream. This has the benefits of avoiding the extra cost of sending data through the Kinesis Firehose + the ability to process the data before sending it to Splunk (or lots of other places)

amiracle
Splunk Employee
Splunk Employee

For Kinesis Firehose, you'll need to have some prerequisites validated prior to sending data into Splunk via Kinesis Firehose.

First, make sure you are using Splunk version 6.6+ . This is required for the HEC health status check. Next, you'll need to have a valid signed SSL certificate on the AWS ELB and a publicly facing IP with sticky sessions enabled. The Splunk Indexers (where the data will be landing from the ELB via HEC) should have the Splunk Add-on for Kinesis Firehose installed and set the stanza ackIdleCleanup = true on the inputs.conf .

Once all that has been done, then you can test your Splunk setup by running the following curl command:

curl https://http-inputs-firehose-<customer>.splunkcloud.com/services/collector/raw?channel=FE0ECFAD-13D5... -H "Authorization: Splunk <HEC_TOKEN>" -d '<raw data string>'

Note that Splunk Cloud does not use the port 8088, but your custom build Splunk instance might.

muebel
SplunkTrust
SplunkTrust

if the splunk instance is 6.7+, there isn't a need for the channel parameter in the POST

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...