Setting up a SH-Cluster with deployer and DMC

[horsefez]

Hi fellow splunkers,

I'm currently setting up a SH-cluster.
I have an indexer-cluster with 2 indexers. One master who is also the deployer. 3 Searchheads in a sh-cluster.

The setup of the indexer-cluster worked fine. I'm able to monitor cluster-status in the dmc on the master.
Setting up the SH-Cluster didn't work out so well sadly. I get the following error trying to distribute shcluster-bundle over the deployer:
In handler 'shclustercaptaincontrol': Search Head Clustering is not enabled on this node. REST endpoint is not available

---

Let me show you some of the configurations first:

The server.conf on all of the three search-heads looks like this:

[general]
serverName = searchhead1
pass4SymmKey = <keyA>

[sslConfig]
sslKeysfilePassword = $1$RMhy1NAQ4pBr

... something something ... 

[replication_port://4444]

[shclustering]
conf_deploy_fetch_url = https://master1:8089
disabled = 0
mgmt_uri = https://searchhead1:8089
pass4SymmKey = <keyB>
shcluster_label = Splunk_Searchheads
id = 6758BAB0-E6C4-4534-ACB1-E71D6A4934B4

[clustering]
master_uri = https://master1:8089
mode = searchhead
pass4SymmKey = <keyB>

The server.conf on the master/deployer looks like this:

serverName = master1
site = default
pass4SymmKey = <keyD>

... something something ...

[clustering]
access_logging_for_heartbeats = 1
cluster_label = Splunk_Indexers
max_peer_build_load = 5
mode = master
pass4SymmKey = <keyB>
replication_factor = 2
buckets_to_summarize = primaries
summary_replication = 0

[sslConfig]
sslKeysfilePassword = <keyC>


[shclustering]
pass4SymmKey = <keyB>
shcluster_label = Splunk_Searchheads

A splunk show shcluster-status on one of the searchheads brings me this:

 Captain:
                          dynamic_captain : 1
                          elected_captain : Mon Jul  4 19:02:06 2016
                                       id : 6758BAB0-E6C4-4534-ACB1-E71D6A4934B4
                         initialized_flag : 1
                                    label : searchhead1
                         maintenance_mode : 0
                                 mgmt_uri : https://searchhead1:8089
                    min_peers_joined_flag : 1
                     rolling_restart_flag : 0
                       service_ready_flag : 1

 Members:
        searchhead1
                                    label : searchhead1
                                 mgmt_uri : https://searchhead1:8089
                           mgmt_uri_alias : https://searchhead1-ip:8089
                                   status : Up
        searchhead3
                                    label : searchhead3
                                 mgmt_uri : https://searchhead3:8089
                           mgmt_uri_alias : https://searchhead3-ip:8089
                                   status : Up
        searchhead2
                                    label : searchhead2
                                 mgmt_uri : https://searchhead2:8089
                           mgmt_uri_alias : https://searchhead2-ip:8089
                                   status : Up

---

On the deployer I get this error when I try to do a splunk show shcluster-status:
In handler 'shclustercaptaincontrol': Search Head Clustering is not enabled on this node. REST endpoint is not available

On the master/deployer I have set up the DMC, but it only shows me the indexers connected to it. No Search-Heads nor the SH-Cluster.

This is weird 😞
I somehow think I forgot to initialize the deployer properly... but then the docs say I should not initialize the deployer like the searchheads.

Any help or suggestions on this are highly apprechiated.

Regards,
pyro_wood

[teunlaan · Accepted solution]

Your "show shcluster-status " looks ok, zo think its an issue in the DMC.

Are the SH's configured as "Search Peer" (settings > distribured search > search peers)? otherwise It can't make rest call's and the won't show up in the DMC

[horsefez]

Thank you!
This was the step I was missing out. Now it works fine!

[gman2015]

Don't think aswer above was really addresing issue with "In handler 'shclustercaptaincontrol': Search Head Clustering is not enabled on this node". It does address issue with pears not showing in DM Console... But not Bundle deployment. Suspect you ran "Show Shcluster status" command on Deployer......

[ddrillic]

You said -

-- On the deployer I get this error when I try to do a splunk show shcluster-status:
In handler 'shclustercaptaincontrol': Search Head Clustering is not enabled on this node. REST endpoint is not available

Why am I getting Search Head Cluster Captain bootstrap error "Raft not initialized"?

says -

The splunk show shcluster-status should not be run on the deployer, it needs to be run on one of the 3 cluster members.

and it explains more -

1. think of the deployer as a separate entity outside the shcluster which needs to be able to communicate with the shcluster (using secret /pass4SymmKey)
2. the init step needs to be run on the 3 members. The init step should leave a [shclustering] stanza in your server.conf with correct settings.
3. Bootstrap then needs to be run on only 1 of the members.
4. As mentioned before the status needs to be checked on only the members.