Indexer Discovery Error; pass4SymmKey or SSL?

[22isaiah]

After setting the pass4SymmKey in my master node's server.conf file and in my forwarder's output.conf file I am still unable to make them communicate for indexer discovery. I made sure I typed the same key in both areas.

#server.conf on master indexer
[general]
serverName = splunk-indexer01
pass4SymmKey = $xxxxxxxxxxxx

[sslConfig]
sslPassword = $xxxxxxxxxxx

[clustering]
pass4SymmKey = $xxxxxxxxxxxxxxxxxxxxxxxxxxxx==
cluster_label = index_cluster
mode = master

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[indexer_discovery]
pass4SymmKey = $xxxxxxxxx=

#output.conf on forwarder
[indexer_discovery:splunk-indexer01]
pass4SymmKey = $xxxxxxxxx=
master_uri = http://10.xxx.xxx.xxx:8089

[tcpout:my_indexers]
indexerDiscovery = splunk-indexer01

[tcpout]
defaultGroup = my_indexers

#errors

Forwarders splunkd.log file

-0700 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:my_indexers] in 'outputs.conf' matches the same setting  under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=http://10.xxx.xxx.xxx:8089/services/indexer_discovery http_code=502 http_response="Connection reset by peer"]

Master indexer's splunkd.log file

-0700 WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

The IPs specified in the error's output are the correct IPs of the master indexer and forwarder, respectively, so they are trying to communicate. I am wondering if the SSL is the real culprit since my indexer discovery is set for tcp, but I'm not sure since I'm getting a pass4SymmKey error and I'm not sure how to solve either of these. Any help would be greatly appreciated. I'm using Splunk Enterprise 7.0.2. Thanks!

[MuS · Accepted solution]

Hi 22isaiah,

but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey 😉

You have this setting on the forwarder in outputs.conf:

master_uri = http://10.130.154.112:8089

but it should be

master_uri = https://10.130.154.112:8089

This is the reason the cluster master is complaining with this message:

WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

So the master is not even checking the pass4Symmkey because the forwarder is not able to establish a proper connection.

Hope this helps ...

cheers, MuS

[deepashri_123]

Hey 22isaiah,

The pass4SymmKey for clustering must be different to indexer_discovery. Try changing password for both stanzas and restart.

[22isaiah]

I set them different to begin with, you can see they are very different in length. Also, I tried changing the indexer discovery password multiple times and rebooting before posting here. I didn't change the cluster password however, because your forwarders don't use that anywhere. Thanks.

[MuS]

Just replaced all passwords with something and cleared the IP.

cheers, MuS

[22isaiah]

I have already tried changing the indexer discovery password and rebooting. Why would I need to change "all passwords" when the forwarder only used the one indexer discovery password? Also, what do you mean by clearing the IP?

[MuS]

This was not an answer to your question: If you include your real encrypted password here, people are still able to decrypt them 😉
That's why I changed/removed them from your post.

Hope this makes sense ...

cheers, MuS