I am trying to figure out how to configure my cluster master to generate a token and HEC configuration information/files to my index cluster. The documentation is not clear as to how this is done. I believe, in the global settings for the token, I can configure the ouptpuGroup with the indexers in my cluster and thereby load-balancing across the bunch of them. Not sure about the configuration needed to do this.
We can create a separate token in master cluster. Copy the configurations and push it to indexers.
Sample configurations.
In mastercluster /opt/splunk/etc/master-apps/http_event_config/local/inputs.conf
[http]
disabled=0
[http://temp]
disabled=0
index = test
source = syslog
token = generated token from mastercluster
Validate and push the config bundle to indexer and test with the below command.
curl -k https://indexerip:8088/services/collector/event -H "Authorization: Splunk XXXXX-generatedtoken-XXXXXX" -d '{"event" : "helloworld"}'
While creating a new HEC token from the master cluster portal, the HEC token generated is located in master cluster VM in the following path= /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf.
How should we push the HEC token from master cluster to the indexer peer using Config bundle action? Should we manually copy the inputs.conf from /opt/splunk/etc/apps/splunk_httpinput/local to /opt/splunk/etc/master-apps/splunk_httpinput/local and then Validate and push the config bundle to indexer?
If you refer to Update common peer configurations and apps you configure the HEC tokens inside the cluster master (or master node) and push the configuration out.
The HEC token is local to each indexer, the indexer receiving the data via HEC will index it, there is no requirement for output groups on an indexer...(nor will it forward to another indexer).
The load balancing of HEC traffic has to be done by something outside the Splunk indexers, for example the client or a load balancer before they get to the indexers on port 8088