Getting Data In

Indexer Discovery Error (IndexerDiscoveryHeartbeatThread)

amitjaywantsplu
Engager

Hi,

I have Splunk 8.0.0 on AWS with a clustered indexer set up (1 Master and 4 indexers) and I have deployed custom test apps (with basic monitoring for windows/Linux logs) on the servers that have the forwarders installed. I have enabled the indexer discovery feature in the outputs.conf file (local folder) for these apps and on the server.conf file of the cluster master (etc/system/local) but I see the following error in the forwarder logs:

04-05-2020 16:57:53.752 +1000 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:target1] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://clustermaster:8089/services/indexer_discovery http_code=502 http_response="Error connecting: Connect Timeout"]

I have ensured that the pass4SymmKey attribute is the same for the outputs.conf on the forwarders and the server.conf on the cluster master (in their respective indexer discovery sections), but yet I see this error.

Any pointers that would help me resolve this?

codebuilder
SplunkTrust
SplunkTrust

The situation you describe generally happens when you configure a forwarder for indexer discovery but provide the hashed pass4SymmKey value from the master, rather that the plain text key.

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

eblair84
Observer

@codebuilder 

I'm (very) new to Splunk. How does one do this:

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

 

Where do I get the non-hashed, plain text key? Also is the forwarder daemon just "splunk" on the forwarder machine?

 

Thanks,

Chris

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...