Security

Indexer Discovery Error; pass4SymmKey or SSL?

22isaiah
New Member

After setting the pass4SymmKey in my master node's server.conf file and in my forwarder's output.conf file I am still unable to make them communicate for indexer discovery. I made sure I typed the same key in both areas.

#server.conf on master indexer
[general]
serverName = splunk-indexer01
pass4SymmKey = $xxxxxxxxxxxx

[sslConfig]
sslPassword = $xxxxxxxxxxx

[clustering]
pass4SymmKey = $xxxxxxxxxxxxxxxxxxxxxxxxxxxx==
cluster_label = index_cluster
mode = master

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[indexer_discovery]
pass4SymmKey = $xxxxxxxxx=

#output.conf on forwarder
[indexer_discovery:splunk-indexer01]
pass4SymmKey = $xxxxxxxxx=
master_uri = http://10.xxx.xxx.xxx:8089

[tcpout:my_indexers]
indexerDiscovery = splunk-indexer01

[tcpout]
defaultGroup = my_indexers

#errors

Forwarders splunkd.log file

-0700 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:my_indexers] in 'outputs.conf' matches the same setting  under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=http://10.xxx.xxx.xxx:8089/services/indexer_discovery http_code=502 http_response="Connection reset by peer"]

Master indexer's splunkd.log file

-0700 WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

The IPs specified in the error's output are the correct IPs of the master indexer and forwarder, respectively, so they are trying to communicate. I am wondering if the SSL is the real culprit since my indexer discovery is set for tcp, but I'm not sure since I'm getting a pass4SymmKey error and I'm not sure how to solve either of these. Any help would be greatly appreciated. I'm using Splunk Enterprise 7.0.2. Thanks!

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi 22isaiah,

but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey 😉

You have this setting on the forwarder in outputs.conf:

master_uri = http://10.130.154.112:8089

but it should be

master_uri = https://10.130.154.112:8089

This is the reason the cluster master is complaining with this message:

WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

So the master is not even checking the pass4Symmkey because the forwarder is not able to establish a proper connection.

Hope this helps ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi 22isaiah,

but now you get an answer 😉
According to the logs it's not related to your pass4SymmKey 😉

You have this setting on the forwarder in outputs.conf:

master_uri = http://10.130.154.112:8089

but it should be

master_uri = https://10.130.154.112:8089

This is the reason the cluster master is complaining with this message:

WARN  HttpListener - Socket error from 10.xxx.xxx.xx while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

So the master is not even checking the pass4Symmkey because the forwarder is not able to establish a proper connection.

Hope this helps ...

cheers, MuS

deepashri_123
Motivator

Hey 22isaiah,

The pass4SymmKey for clustering must be different to indexer_discovery. Try changing password for both stanzas and restart.

0 Karma

22isaiah
New Member

I set them different to begin with, you can see they are very different in length. Also, I tried changing the indexer discovery password multiple times and rebooting before posting here. I didn't change the cluster password however, because your forwarders don't use that anywhere. Thanks.

0 Karma

MuS
SplunkTrust
SplunkTrust

Just replaced all passwords with something and cleared the IP.

cheers, MuS

0 Karma

22isaiah
New Member

I have already tried changing the indexer discovery password and rebooting. Why would I need to change "all passwords" when the forwarder only used the one indexer discovery password? Also, what do you mean by clearing the IP?

0 Karma

MuS
SplunkTrust
SplunkTrust

This was not an answer to your question: If you include your real encrypted password here, people are still able to decrypt them 😉
That's why I changed/removed them from your post.

Hope this makes sense ...

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...