Deployment Architecture

scaling HEC on heavy forwarders

raindrop18
Communicator

currently I am using single heavy forwarders as my HEC and the token generated from one heavy forwarders, however to avoid single point of failure I want expand to other 3 heavy forwarders but as the token generated from one single server how to use that one token to the rest for forwarder. any feedback highly appreciate it.

thanks

Tags (1)
0 Karma
1 Solution

nickhills
Ultra Champion

You can manually create inputs on your HFs (or build an app to deploy them) and set the token across your environment.

In inputs.conf you need:

[http://your input name]
disabled = 0
token =your_token
<any other settings>
If my comment helps, please give it a thumbs up!

View solution in original post

fabiocaldas
Contributor

Hi raindrop18,

I'm using a cluster of Heavy Forwarders as HEC endpoints and I'm controlling it from master. I set my Heavy Forwarder to be a deployment client and I distribute HEC token from master to all of them !!

0 Karma

nickhills
Ultra Champion

Hi @fabiocaldas,
Do you mean your cluster master is also a deployment server?
(I note that this is not a supported configuration, even though it works)

I hate to be pedantic, but in Splunk terminonology ‘cluster’ has a specific meaning. You can have a team, group, collection or even a pool of heavy forwarders, but you should avoid using the term ‘cluster’ to avoid confusion.

I think what your saying is your ‘pool’ of heavy forwarders are deployment clients of your deployment server.

The fact that your DS may (or may not be) collocated with your “indexer cluster master” is immaterial, but I would caution others from doing so.

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

You can manually create inputs on your HFs (or build an app to deploy them) and set the token across your environment.

In inputs.conf you need:

[http://your input name]
disabled = 0
token =your_token
<any other settings>
If my comment helps, please give it a thumbs up!

raindrop18
Communicator

thanks much for response, do you have a doc link for the setup. I like to see the other settiing like index so on. appreciate it again.

0 Karma

nickhills
Ultra Champion

Side note: Although the Splunk UI will generate you a 'token' you can use any valid 128 bit GUID/UUID which you can generate outside of Splunk, there is no 'magic' in the value Splunk generates for you!
https://www.guidgenerator.com/

If my comment helps, please give it a thumbs up!
0 Karma

raindrop18
Communicator

thanks, last question is that possible to create multiple endpoint and token in one input.conf? I have multiple HEC endpoint pointing different index and source type. one input.conf can handle that or I need multiple input.conf.

0 Karma

nickhills
Ultra Champion

That’s fine, you can load them all up in one file, as long as the input names and tokens are unique.

If my comment helps, please give it a thumbs up!
0 Karma

raindrop18
Communicator

appreciate the quick response. is this works

[http]
index = A
sourcetype = log4j
disabled = 0

[http://collector-A]
disabled = 0
host = splunk-server
index = my-test1
indexes = my-test1
source = test1
token = XXXXXXXXXXXXXXXXXXXXXX
sourcetype = log4j


[http]
index = B
sourcetype = log4j
disabled = 0

[http://collector-B]
disabled = 0
host = splunk-server
index = my-test2
indexes = my-test2
source = test2
token = XXXXXXXXXXXXXXXXXXXXXX
sourcetype = log4j
0 Karma

nickhills
Ultra Champion

Apologies for poor formatting. (I am on a phone) but this should be all you need.

 [http://collector-A]
 disabled = 0
 host = splunk-server
 index = my-test1
 source = test1
 token = XXXXXXXXXXXXXXXXXXXXXX
 sourcetype = log4j

 [http://collector-B]
 disabled = 0
 host = splunk-server
index = my-test2
 source = test2
 token = XXXXXXXXXXXXXXXXXXXXXX

 sourcetype = log4j
If my comment helps, please give it a thumbs up!
0 Karma

raindrop18
Communicator

thank you so very much!!!!

0 Karma

nickhills
Ultra Champion

Sure: http://docs.splunk.com/Documentation/Splunk/7.0.1/admin/Inputsconf#http:_.28HTTP_Event_Collector.29

that covers all input parameters across all input types, but it will take you straight to the HEC specific settings to start.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...