Splunk Enterprise

Unable to add stand alone search head to Indexer Cluster

sesharao92
Explorer

I want to create a new search apart from the existing searchhead cluster.
I have added the following configuration into server.conf. But the connection between search head and master node is failing.

[clustering]
pass4SymmKey = xxxx (copied from existing SHC)
mode = searchhead
master_uri = https://:8089
multisite = true

Error:
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Can I create separate searchhead and configure the master node along with the existing SHC?

Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you added pass4SymmKey in server.conf in plain text format ? If you just copy and paste pass4SymmKey from another server then it will not work because it is encrypted.

View solution in original post

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you added pass4SymmKey in server.conf in plain text format ? If you just copy and paste pass4SymmKey from another server then it will not work because it is encrypted.

0 Karma

sesharao92
Explorer

I copied from existing shcluser search head and pasted into the newly created search head. I will add the actual pass4SymmKey and test it..

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If you don't know decrypted key then you can decrypt it, reference doc https://www.hurricanelabs.com/splunk-tutorials/make-splunk-do-it-how-to-decrypt-passwords-encrypted-... or if it is fresh installation then I'll suggest to copy $SPLUNK_HOME/etc/auth/splunk.secret from existing SHC and place it in new server but this might create problem because few of the default password already encrypted when you start splunk so I suggest to follow document from Hurrican Labs.

0 Karma

sesharao92
Explorer

Thanks for the help.. I was able to decrypt the key and able to add search head to the cluster..

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Converted my comment to answer so you can accept it.

0 Karma

dkeck
Influencer

HI,

did you try to add the search head peer via UI?

0 Karma

sesharao92
Explorer

I tried to add the master node with the UI. It's giving the same error..

0 Karma

sesharao92
Explorer

I can able to add search peers, but unable to add master node.
Do I need to add search peers separately to the newly created search head. I thought adding master node will be sufficient.
I had 12 search peer nodes and a master node. I am trying to add master node to the newly created search head. But it's failing with below error.
Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

0 Karma

dkeck
Influencer

sry I don´t get what you mean by "adding a master node to a search head".

You can add a seach head to be a SH in a cluster, so you would add this search head to the cluster.

is this what you mean?

0 Karma

sesharao92
Explorer

yes. I tried to configure search head in the cluster. While configuring it was asked for master node uri.. I gave it. But i got the above error.

0 Karma

dkeck
Influencer

Ok do you see any errors in the splunkd.log of both server? might be a hint in there.

Does the communication between both is working on Port 8089? Mabye firewall is blocking it

0 Karma

sesharao92
Explorer

I can able to connect to the server using 8089 port..

0 Karma

sesharao92
Explorer

01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Checking for localhost key pair
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-21-2019 22:43:32.021 +1100 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.022 +1100 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-21-2019 22:43:32.677 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.159 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.
01-21-2019 22:43:33.581 +1100 ERROR ApplicationUpdater - Error checking for update, URL=https://apps.splunk.com/api/apps:resolve/checkforupgrade: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
01-21-2019 22:44:40.629 +1100 ERROR ClusterStatusHandler - Could not contact master. Check that the master is up, the master_uri=https://:8089 and secret are specified correctly

Check these logs..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...