We're trying to send data to a our single Heavy Forwarder via the new destination type of Splunk in Kinesis Firehose. We've configured the Firehose to send data as "event" type (rather than "raw") since we use a transformation AWS Lambda to configure the http requests body as JSON (as described in Format HEC events documentation) but for some reason, Firehose fails with reason:
The connection to Splunk timed out. This might be a transient error and the request will be retried. Kinesis Firehose backs up the data to Amazon S3 if all retries fail
But when I try to curl to the HEC endpoint from my laptop, or any other network, with the relevant token, I see messages in Splunk and flow is valid.
Any suggestions? Where at least can I see logging for HEC to verify that requests arrive to Splunk and see their error message? (and not just metrics...)
I'm using version Splunk Enterprise 6.5.1 with AWS Kinesis Firehose Add-on for Splunk
... View more