Splunk Enterprise

Unable to add stand alone search head to Indexer Cluster

sesharao92
Explorer

I want to create a new search apart from the existing searchhead cluster.
I have added the following configuration into server.conf. But the connection between search head and master node is failing.

[clustering]
pass4SymmKey = xxxx (copied from existing SHC)
mode = searchhead
master_uri = https://:8089
multisite = true

Error:
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Can I create separate searchhead and configure the master node along with the existing SHC?

Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you added pass4SymmKey in server.conf in plain text format ? If you just copy and paste pass4SymmKey from another server then it will not work because it is encrypted.

View solution in original post

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you added pass4SymmKey in server.conf in plain text format ? If you just copy and paste pass4SymmKey from another server then it will not work because it is encrypted.

0 Karma

sesharao92
Explorer

I copied from existing shcluser search head and pasted into the newly created search head. I will add the actual pass4SymmKey and test it..

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If you don't know decrypted key then you can decrypt it, reference doc https://www.hurricanelabs.com/splunk-tutorials/make-splunk-do-it-how-to-decrypt-passwords-encrypted-... or if it is fresh installation then I'll suggest to copy $SPLUNK_HOME/etc/auth/splunk.secret from existing SHC and place it in new server but this might create problem because few of the default password already encrypted when you start splunk so I suggest to follow document from Hurrican Labs.

0 Karma

sesharao92
Explorer

Thanks for the help.. I was able to decrypt the key and able to add search head to the cluster..

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Converted my comment to answer so you can accept it.

0 Karma

dkeck
Influencer

HI,

did you try to add the search head peer via UI?

0 Karma

sesharao92
Explorer

I tried to add the master node with the UI. It's giving the same error..

0 Karma

sesharao92
Explorer

I can able to add search peers, but unable to add master node.
Do I need to add search peers separately to the newly created search head. I thought adding master node will be sufficient.
I had 12 search peer nodes and a master node. I am trying to add master node to the newly created search head. But it's failing with below error.
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

0 Karma

dkeck
Influencer

sry I don´t get what you mean by "adding a master node to a search head".

You can add a seach head to be a SH in a cluster, so you would add this search head to the cluster.

is this what you mean?

0 Karma

sesharao92
Explorer

yes. I tried to configure search head in the cluster. While configuring it was asked for master node uri.. I gave it. But i got the above error.

0 Karma

dkeck
Influencer

Ok do you see any errors in the splunkd.log of both server? might be a hint in there.

Does the communication between both is working on Port 8089? Mabye firewall is blocking it

0 Karma

sesharao92
Explorer

I can able to connect to the server using 8089 port..

0 Karma

sesharao92
Explorer

01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Checking for localhost key pair
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.022 +1100 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.677 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.159 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 ERROR ApplicationUpdater - Error checking for update, URL=https://apps.splunk.com/api/apps:resolve/checkforupgrade: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
01-21-2019 22:44:40.629 +1100 ERROR ClusterStatusHandler - Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Check these logs..

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...