All Apps and Add-ons

HEC integration with AWS Kinesis Firehose



We're trying to send data to a our single Heavy Forwarder via the new destination type of Splunk in Kinesis Firehose. We've configured the Firehose to send data as "event" type (rather than "raw") since we use a transformation AWS Lambda to configure the http requests body as JSON (as described in Format HEC events documentation) but for some reason, Firehose fails with reason:

The connection to Splunk timed out. This might be a transient error and the request will be retried. Kinesis Firehose backs up the data to Amazon S3 if all retries fail
But when I try to curl to the HEC endpoint from my laptop, or any other network, with the relevant token, I see messages in Splunk and flow is valid.

Any suggestions? Where at least can I see logging for HEC to verify that requests arrive to Splunk and see their error message? (and not just metrics...)

I'm using version Splunk Enterprise 6.5.1 with AWS Kinesis Firehose Add-on for Splunk

0 Karma

Splunk Employee
Splunk Employee

Hi there,

AWS Kinesis Firehose to Splunk requires Splunk version 6.6 or higher.

For future reference, there is an HEC dashboard in the Monitoring Console that shows a summary of activity and errors. In Splunk Enterprise, it's found under Settings -> Monitoring Console -> Indexing -> Inputs -> HTTP Event Collector.

HEC logs are in the splunkd logs found in index=_internal (filter by source for the splunkd logs on disk). Some keywords to search for include HttpEventCollector, HttpInputDataHandler, HttpInputEventParser, HttpInputQueue, and HttpInputServer. You can enable different logging levels for these components under Settings -> Server Settings -> Server Logging.

Hope this helps!


.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!