- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

After setting the pass4SymmKey in my master node's server.conf file and in my forwarder's output.conf file I am still unable to make them communicate for indexer discovery. I made sure I typed the same key in both areas.
#server.conf on master indexer
[general]
serverName = splunk-indexer01
pass4SymmKey = $xxxxxxxxxxxx
[sslConfig]
sslPassword = $xxxxxxxxxxx
[clustering]
pass4SymmKey = $xxxxxxxxxxxxxxxxxxxxxxxxxxxx==
cluster_label = index_cluster
mode = master
[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[indexer_discovery]
pass4SymmKey = $xxxxxxxxx=
#output.conf on forwarder
[indexer_discovery:splunk-indexer01]
pass4SymmKey = $xxxxxxxxx=
master_uri = http://10.xxx.xxx.xxx:8089
[tcpout:my_indexers]
indexerDiscovery = splunk-indexer01
[tcpout]
defaultGroup = my_indexers
#errors
Forwarders splunkd.log file
-0700 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:my_indexers] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=http://10.xxx.xxx.xxx:8089/services/indexer_discovery http_code=502 http_response="Connection reset by peer"]
Master indexer's splunkd.log file
-0700 WARN HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
The IPs specified in the error's output are the correct IPs of the master indexer and forwarder, respectively, so they are trying to communicate. I am wondering if the SSL is the real culprit since my indexer discovery is set for tcp, but I'm not sure since I'm getting a pass4SymmKey error and I'm not sure how to solve either of these. Any help would be greatly appreciated. I'm using Splunk Enterprise 7.0.2. Thanks!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hi 22isaiah,
but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey
😉
You have this setting on the forwarder in outputs.conf
:
master_uri = http://10.130.154.112:8089
but it should be
master_uri = https://10.130.154.112:8089
This is the reason the cluster master is complaining with this message:
WARN HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
So the master is not even checking the pass4Symmkey
because the forwarder is not able to establish a proper connection.
Hope this helps ...
cheers, MuS
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hi 22isaiah,
but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey
😉
You have this setting on the forwarder in outputs.conf
:
master_uri = http://10.130.154.112:8089
but it should be
master_uri = https://10.130.154.112:8089
This is the reason the cluster master is complaining with this message:
WARN HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
So the master is not even checking the pass4Symmkey
because the forwarder is not able to establish a proper connection.
Hope this helps ...
cheers, MuS
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey 22isaiah,
The pass4SymmKey for clustering must be different to indexer_discovery. Try changing password for both stanzas and restart.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

I set them different to begin with, you can see they are very different in length. Also, I tried changing the indexer discovery password multiple times and rebooting before posting here. I didn't change the cluster password however, because your forwarders don't use that anywhere. Thanks.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Just replaced all passwords with something
and cleared the IP.
cheers, MuS
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

I have already tried changing the indexer discovery password and rebooting. Why would I need to change "all passwords" when the forwarder only used the one indexer discovery password? Also, what do you mean by clearing the IP?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

This was not an answer to your question: If you include your real encrypted password here, people are still able to decrypt them 😉
That's why I changed/removed them from your post.
Hope this makes sense ...
cheers, MuS
