Archive

Indexer Discovery Error; pass4SymmKey or SSL?

New Member

After setting the pass4SymmKey in my master node's server.conf file and in my forwarder's output.conf file I am still unable to make them communicate for indexer discovery. I made sure I typed the same key in both areas.

#server.conf on master indexer
[general]
serverName = splunk-indexer01
pass4SymmKey = $xxxxxxxxxxxx

[sslConfig]
sslPassword = $xxxxxxxxxxx

[clustering]
pass4SymmKey = $xxxxxxxxxxxxxxxxxxxxxxxxxxxx==
cluster_label = index_cluster
mode = master

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[indexer_discovery]
pass4SymmKey = $xxxxxxxxx=

#output.conf on forwarder
[indexer_discovery:splunk-indexer01]
pass4SymmKey = $xxxxxxxxx=
master_uri = http://10.xxx.xxx.xxx:8089

[tcpout:my_indexers]
indexerDiscovery = splunk-indexer01

[tcpout]
defaultGroup = my_indexers

#errors

Forwarders splunkd.log file

-0700 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:my_indexers] in 'outputs.conf' matches the same setting  under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=http://10.xxx.xxx.xxx:8089/services/indexer_discovery http_code=502 http_response="Connection reset by peer"]

Master indexer's splunkd.log file

-0700 WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

The IPs specified in the error's output are the correct IPs of the master indexer and forwarder, respectively, so they are trying to communicate. I am wondering if the SSL is the real culprit since my indexer discovery is set for tcp, but I'm not sure since I'm getting a pass4SymmKey error and I'm not sure how to solve either of these. Any help would be greatly appreciated. I'm using Splunk Enterprise 7.0.2. Thanks!

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi 22isaiah,

but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey 😉

You have this setting on the forwarder in outputs.conf:

master_uri = http://10.130.154.112:8089

but it should be

master_uri = https://10.130.154.112:8089

This is the reason the cluster master is complaining with this message:

WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

So the master is not even checking the pass4Symmkey because the forwarder is not able to establish a proper connection.

Hope this helps ...

cheers, MuS

View solution in original post

SplunkTrust
SplunkTrust

Hi 22isaiah,

but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey 😉

You have this setting on the forwarder in outputs.conf:

master_uri = http://10.130.154.112:8089

but it should be

master_uri = https://10.130.154.112:8089

This is the reason the cluster master is complaining with this message:

WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

So the master is not even checking the pass4Symmkey because the forwarder is not able to establish a proper connection.

Hope this helps ...

cheers, MuS

View solution in original post

Motivator

Hey 22isaiah,

The pass4SymmKey for clustering must be different to indexer_discovery. Try changing password for both stanzas and restart.

0 Karma

New Member

I set them different to begin with, you can see they are very different in length. Also, I tried changing the indexer discovery password multiple times and rebooting before posting here. I didn't change the cluster password however, because your forwarders don't use that anywhere. Thanks.

0 Karma

SplunkTrust
SplunkTrust

Just replaced all passwords with something and cleared the IP.

cheers, MuS

0 Karma

New Member

I have already tried changing the indexer discovery password and rebooting. Why would I need to change "all passwords" when the forwarder only used the one indexer discovery password? Also, what do you mean by clearing the IP?

0 Karma

SplunkTrust
SplunkTrust

This was not an answer to your question: If you include your real encrypted password here, people are still able to decrypt them 😉
That's why I changed/removed them from your post.

Hope this makes sense ...

cheers, MuS

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!