All Apps and Add-ons

HEC integration with AWS Kinesis Firehose

sundaysky
Engager

Hi,

We're trying to send data to a our single Heavy Forwarder via the new destination type of Splunk in Kinesis Firehose. We've configured the Firehose to send data as "event" type (rather than "raw") since we use a transformation AWS Lambda to configure the http requests body as JSON (as described in Format HEC events documentation) but for some reason, Firehose fails with reason:

The connection to Splunk timed out. This might be a transient error and the request will be retried. Kinesis Firehose backs up the data to Amazon S3 if all retries fail
But when I try to curl to the HEC endpoint from my laptop, or any other network, with the relevant token, I see messages in Splunk and flow is valid.

Any suggestions? Where at least can I see logging for HEC to verify that requests arrive to Splunk and see their error message? (and not just metrics...)

I'm using version Splunk Enterprise 6.5.1 with AWS Kinesis Firehose Add-on for Splunk
Thanks.

0 Karma

eprokop_splunk
Splunk Employee
Splunk Employee

Hi there,

AWS Kinesis Firehose to Splunk requires Splunk version 6.6 or higher.

For future reference, there is an HEC dashboard in the Monitoring Console that shows a summary of activity and errors. In Splunk Enterprise, it's found under Settings -> Monitoring Console -> Indexing -> Inputs -> HTTP Event Collector.

HEC logs are in the splunkd logs found in index=_internal (filter by source for the splunkd logs on disk). Some keywords to search for include HttpEventCollector, HttpInputDataHandler, HttpInputEventParser, HttpInputQueue, and HttpInputServer. You can enable different logging levels for these components under Settings -> Server Settings -> Server Logging.

Hope this helps!

Eric

Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...