Getting Data In

Indexer Discovery Error (IndexerDiscoveryHeartbeatThread)

amitjaywantsplu
Engager

Hi,

I have Splunk 8.0.0 on AWS with a clustered indexer set up (1 Master and 4 indexers) and I have deployed custom test apps (with basic monitoring for windows/Linux logs) on the servers that have the forwarders installed. I have enabled the indexer discovery feature in the outputs.conf file (local folder) for these apps and on the server.conf file of the cluster master (etc/system/local) but I see the following error in the forwarder logs:

04-05-2020 16:57:53.752 +1000 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:target1] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://clustermaster:8089/services/indexer_discovery http_code=502 http_response="Error connecting: Connect Timeout"]

I have ensured that the pass4SymmKey attribute is the same for the outputs.conf on the forwarders and the server.conf on the cluster master (in their respective indexer discovery sections), but yet I see this error.

Any pointers that would help me resolve this?

codebuilder
SplunkTrust
SplunkTrust

The situation you describe generally happens when you configure a forwarder for indexer discovery but provide the hashed pass4SymmKey value from the master, rather that the plain text key.

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

eblair84
Observer

@codebuilder 

I'm (very) new to Splunk. How does one do this:

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

 

Where do I get the non-hashed, plain text key? Also is the forwarder daemon just "splunk" on the forwarder machine?

 

Thanks,

Chris

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...