Are there any subtleties to setting up a search head cluster in docker containers that could result in a Search Head Clustering(not enabled on this node)?

[dtow1]

Hello,

I have seen how common this issue seems to be, but have not been able to get my search head cluster to set up correctly. I've tried rebuilding from scratch and starting off by initializing the search heads through the GUI in case I was doing the commands incorrectly. I can't seem to get around the error in the title. "Search Head Clustering is not enabled on this node. REST endpoint is not available" when I run /opt/splunk/bin/splunk show shcluster-status. I am also unable to elect a captain.

My setup is using docker containers. 1 Masternode, also my deployer, 2 peers, 3 search heads. All three show up with a status of UP when I go into Settings -> Index Clustering -> Search Heads.

I have three search heads that have the following identical stanzas in /opt/splunk/etc/system/local/server.conf:

[general]
serverName = cb053c44cf2a
pass4SymmKey = $1$1Ewu7tzuOCNj

[sslConfig]
sslPassword = $1$ghWz79vwckt2

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[replication_port://9887]

[shclustering]
conf_deploy_fetch_url = https://172.17.0.2:8089
shcluster_label = shcluster1

[clustering]
master_uri = clustermaster:172.17.0.2:8089
mode = searchhead

[clustermaster:172.17.0.2:8089]
master_uri = https://172.17.0.2:8089
multisite = false
pass4SymmKey = $1$/FPppMjqbE0Wcw==
site = default

Here is my server.conf for the master node.

general]
serverName = 47e8780df786
pass4SymmKey = $1$1Ewu7tzuOCNj

[sslConfig]
sslPassword = $1$gwB6spmoc3Fj

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[clustering]
cluster_label = shcluster1
mode = master
pass4SymmKey = $1$/UYg+YqybXcDDg==
replication_factor = 2

[shclustering]
pass4SymmKey = $1$FUz82GxiRfklJw==
shcluster_label = shcluster1

Thanks for any help.

[dtow1]

I think unsurprisingly the issue here was a PEBKAC error. I was able to get them to check in using the following:

/opt/splunk/bin/splunk init shcluster-config -auth admin:PASSWORD -mgmt_uri https://INTERNAL_DOCKER_URL_FOR_EACH_SEARCHHEAD:8089 -replication_port 9887 -replication_factor 2 -conf_deploy_fetch_url https://172.31.30.232:8089 -secret admin:PASSWORD -shcluster_label shcluster1

I ran this on each box and they are now working and I was able to elect a captain. When I have a little time I will try and identify what my error was (unless someone else does first), and upload the new working server.conf

[s2_splunk]

Just two comments, really not too closely related to your issue... 😉

1. Running Splunk Enterprise in Docker is not currently supported. Just be aware of that as it likely means you will have trouble getting support from Splunk if something doesn't work as it should. Obviously not that relevant if this is just a lab environment/setup
2. You have a top-heavy environment with three search heads going against just two search peers that - under full search load - will likely overwhelm your search peers. Again, if this is a lab setup, it may not matter.

Just thought I'd mention these two things, fwiw.

[dtow1]

Hi Sorry, I meant to reply sooner. I really appreciate those comments. It is just a "throw away" test environment so there isn't actually any traffic being generated. It is good to know both of those points though when applying what I do in my test environment over to real scenarios.