Unable to add stand alone search head to Indexer Cluster

[sesharao92]

I want to create a new search apart from the existing searchhead cluster.
I have added the following configuration into server.conf. But the connection between search head and master node is failing.

[clustering]
pass4SymmKey = xxxx (copied from existing SHC)
mode = searchhead
master_uri = https://:8089
multisite = true

Error:
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Can I create separate searchhead and configure the master node along with the existing SHC?

[harsmarvania57 · Accepted solution]

Hi,

Have you added pass4SymmKey in server.conf in plain text format ? If you just copy and paste pass4SymmKey from another server then it will not work because it is encrypted.

[sesharao92]

I copied from existing shcluser search head and pasted into the newly created search head. I will add the actual pass4SymmKey and test it..

[harsmarvania57]

If you don't know decrypted key then you can decrypt it, reference doc https://www.hurricanelabs.com/splunk-tutorials/make-splunk-do-it-how-to-decrypt-passwords-encrypted-... or if it is fresh installation then I'll suggest to copy $SPLUNK_HOME/etc/auth/splunk.secret from existing SHC and place it in new server but this might create problem because few of the default password already encrypted when you start splunk so I suggest to follow document from Hurrican Labs.

[sesharao92]

Thanks for the help.. I was able to decrypt the key and able to add search head to the cluster..

[harsmarvania57]

Converted my comment to answer so you can accept it.

[dkeck]

HI,

did you try to add the search head peer via UI?

[sesharao92]

I tried to add the master node with the UI. It's giving the same error..

[sesharao92]

I can able to add search peers, but unable to add master node.
Do I need to add search peers separately to the newly created search head. I thought adding master node will be sufficient.
I had 12 search peer nodes and a master node. I am trying to add master node to the newly created search head. But it's failing with below error.
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

[dkeck]

sry I don´t get what you mean by "adding a master node to a search head".

You can add a seach head to be a SH in a cluster, so you would add this search head to the cluster.

is this what you mean?

[sesharao92]

yes. I tried to configure search head in the cluster. While configuring it was asked for master node uri.. I gave it. But i got the above error.

[dkeck]

Ok do you see any errors in the splunkd.log of both server? might be a hint in there.

Does the communication between both is working on Port 8089? Mabye firewall is blocking it

[sesharao92]

I can able to connect to the server using 8089 port..

[sesharao92]

01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Checking for localhost key pair
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.022 +1100 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.677 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.159 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 ERROR ApplicationUpdater - Error checking for update, URL=https://apps.splunk.com/api/apps:resolve/checkforupgrade: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
01-21-2019 22:44:40.629 +1100 ERROR ClusterStatusHandler - Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Check these logs..