Splunk Enterprise

set-up hec behind load balancer

test_splunk15
Explorer

Hi,

we have a requirement where we need to send data from kinesis streams to Splunk via firehose using hec tokens.

In this use case I need to set-up infra in such a way that firehose should be configured with load balancer which is behind heavy forwarders (if one goes down other should be picked-up) -> which will then sends data to indexers.

As hec tokens are created individually on heavy forwarders, if we configure one hec on firehose and load balacer url there it will not be working, what is the best way to configure load balancer for firehose along with the tokens generated (we dont have cluster at HF).

referred below but they are not for firehose to HF load balancers 

https://docs.aws.amazon.com/firehose/latest/dev/writing-with-kinesis-streams.html

https://www.splunk.com/en_us/blog/cloud/power-data-ingestion-into-splunk-using-amazon-kinesis-data-f...

Let me know if you need more info.

 

Thanks

Labels (1)
Tags (1)
0 Karma

livehybrid
Contributor

Hi @test_splunk15 ,

It sounds like you're generating unique HEC tokens on each indexer? If you want to be able to loadbalance your HEC then you will need the same set of tokens on each indexer.

If you're using a cluster then you can push these out using your cluster master (create an app in $SPLUNK_HOME/etc/master-apps containing the inputs.conf with your tokens.
If you're not using a cluster master then you could deploy out with a deployment server, and if not then you can manually copy the tokens between your indexers.

Once done you'll be able to loadbalance your HEC and the token will be accepted on any of the targets the LB hits.

I hope this helps, shout if you have any other questions

Thanks
Will 

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...