All Apps and Add-ons

Why can't Kinesis Firehose reach my heavy forwarder?

ejahnke
Explorer

Hi,

we're trying to get cloudwatchlogs via Kinesis Firehose to a heavy forwarder in a VPC. It's not working currently, we're getting the error:
"Destination: https://XX.XX.XX.XX:8088 - Failed to deliver data to Splunk or to receive acknowledgment. Make sure HEC endpoint is reachable from Firehose and it is healthy."

Does anyone has some ideas?

Regards and thanks in advance,

Eric

0 Karma
1 Solution

amiracle
Splunk Employee
Splunk Employee

Eric,

It looks like the issue with your deployment will be the self-signed cert, you need a third party cert on either the ELB or the HF itself. Also, this is usually done behind an ELB with sticky-bits enabled. Here is a document I put together that might be able to help you: https://github.com/amiracle/cooking_with_Splunk_and_AWS/blob/master/09%20-%20Setting%20up%20Kinesis%...

View solution in original post

amiracle
Splunk Employee
Splunk Employee

Eric,

It looks like the issue with your deployment will be the self-signed cert, you need a third party cert on either the ELB or the HF itself. Also, this is usually done behind an ELB with sticky-bits enabled. Here is a document I put together that might be able to help you: https://github.com/amiracle/cooking_with_Splunk_and_AWS/blob/master/09%20-%20Setting%20up%20Kinesis%...

View solution in original post

ejahnke
Explorer

Thanks for the input. I'm gonna try it.

0 Karma

ejahnke
Explorer

One quick question: You say it will work if we put the cert. on the HF without using an ELB.
Does it really? Or do we HAVE to use an ELB, if we're in a VPC?
The docs don't state if it will work:
"If your indexers are in an AWS Virtual Private Cloud, send your Amazon Kinesis Firehose data to an Elastic Load Balancer (ELB) with sticky sessions enabled and cookie expiration disabled. Follow the directions on this page to configure an ELB that can integrate with the Splunk HTTP event collector." (https://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureanELB)

0 Karma

amiracle
Splunk Employee
Splunk Employee

While it should work with the sending directly to the HF, you'll need to make sure that you've enabled HEC acknowledgement and set ackIdleCleanup = true in inputs.conf . Generally, when you're dealing with VPCFlow or other high volume inputs, you're going to want to use an ELB to spread the load across your HF or Indexing tier. This could lead to hot spotting on your indexers and overall poor search performance.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!