HEC integration with AWS Kinesis Firehose

[sundaysky]

Hi,

We're trying to send data to a our single Heavy Forwarder via the new destination type of Splunk in Kinesis Firehose. We've configured the Firehose to send data as "event" type (rather than "raw") since we use a transformation AWS Lambda to configure the http requests body as JSON (as described in Format HEC events documentation) but for some reason, Firehose fails with reason:

The connection to Splunk timed out. This might be a transient error and the request will be retried. Kinesis Firehose backs up the data to Amazon S3 if all retries fail
But when I try to curl to the HEC endpoint from my laptop, or any other network, with the relevant token, I see messages in Splunk and flow is valid.

Any suggestions? Where at least can I see logging for HEC to verify that requests arrive to Splunk and see their error message? (and not just metrics...)

I'm using version Splunk Enterprise 6.5.1 with AWS Kinesis Firehose Add-on for Splunk
Thanks.

[eprokop_splunk]

Hi there,

AWS Kinesis Firehose to Splunk requires Splunk version 6.6 or higher.

For future reference, there is an HEC dashboard in the Monitoring Console that shows a summary of activity and errors. In Splunk Enterprise, it's found under Settings -> Monitoring Console -> Indexing -> Inputs -> HTTP Event Collector.

HEC logs are in the splunkd logs found in index=_internal (filter by source for the splunkd logs on disk). Some keywords to search for include HttpEventCollector, HttpInputDataHandler, HttpInputEventParser, HttpInputQueue, and HttpInputServer. You can enable different logging levels for these components under Settings -> Server Settings -> Server Logging.

Hope this helps!

Eric