Why can't Kinesis Firehose reach my heavy forwarder?

[ejahnke]

Hi,

we're trying to get cloudwatchlogs via Kinesis Firehose to a heavy forwarder in a VPC. It's not working currently, we're getting the error:
"Destination: https://XX.XX.XX.XX:8088 - Failed to deliver data to Splunk or to receive acknowledgment. Make sure HEC endpoint is reachable from Firehose and it is healthy."

• We confirmed that the HEC-Endpoint is working via curl. (ACK enabled)
• We opened to the CIDR-Blocks as described here: https://docs.aws.amazon.com/de_de/firehose/latest/dev/controlling-access.html#using-iam-splunk-vpc
• Our HF is on Splunk 7.1.1. It's SSL secured with the shipped cert. from splunk. Maybe that's the problem? (as described here: http://docs.splunk.com/Documentation/AddOns/released/Firehose/Troubleshoot#SSL-related_data_delivery...)
• Required ports are open (8088, 8089).
• We aren't getting any info regarding the HEC-Endpoint in the internal logs.
• Other HEC-Endpoints on the HF are working fine.

Does anyone has some ideas?

Regards and thanks in advance,

Eric

[amiracle · Accepted solution]

Eric,

It looks like the issue with your deployment will be the self-signed cert, you need a third party cert on either the ELB or the HF itself. Also, this is usually done behind an ELB with sticky-bits enabled. Here is a document I put together that might be able to help you: https://github.com/amiracle/cooking_with_Splunk_and_AWS/blob/master/09%20-%20Setting%20up%20Kinesis%...

[ejahnke]

Thanks for the input. I'm gonna try it.

[ejahnke]

One quick question: You say it will work if we put the cert. on the HF without using an ELB.
Does it really? Or do we HAVE to use an ELB, if we're in a VPC?
The docs don't state if it will work:
"If your indexers are in an AWS Virtual Private Cloud, send your Amazon Kinesis Firehose data to an Elastic Load Balancer (ELB) with sticky sessions enabled and cookie expiration disabled. Follow the directions on this page to configure an ELB that can integrate with the Splunk HTTP event collector." (https://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureanELB)

[amiracle]

While it should work with the sending directly to the HF, you'll need to make sure that you've enabled HEC acknowledgement and set ackIdleCleanup = true in inputs.conf . Generally, when you're dealing with VPCFlow or other high volume inputs, you're going to want to use an ELB to spread the load across your HF or Indexing tier. This could lead to hot spotting on your indexers and overall poor search performance.