Deployment Architecture

Docker Start Breaks Cluster

New Member

As we have no dev environment I have tried to learn Terraform and Ansible and build my own on Docker.

I now have 2 x Search heads in a cluster, 2 Indexers and an Indexer cluster master, 1 x heavy forwarder, 1 combined deployer/deployment server and a Universal forwarder.

Everything works fine and I can build the whole environment in a few minutes.

But if I stop the containers when I do a "docker start" the cluster configuration of the indexer cluster master and deployer are reset back to the default. 

This is the shclustering stanza of server.conf on the deployer when the environment is built:

[shclustering]
pass4SymmKey = $7$P6EHXzK5D7eS/B6970mBtVsoThkdIn27+xiyZdy2tkOAveg1O3o2rg==
shcluster_label = shcluster_label

And this is after the docker start:

[shclustering]
pass4SymmKey =
shcluster_label = shc_label

This is the clustering stanza from the indexer cluster master server.conf initially:

[clustering]
cluster_label = idxcluster_label
mode = master
search_factor = 1
pass4SymmKey = $7$WLLkzIXVZZmbtPcy1YDkhUNyKI1mzMMPz2Q0dTbivBHxFAokebPZose71eiT
replication_factor = 1

And this is after the docker start:

[clustering]
cluster_label = idxc_label
mode = master
search_factor = 3
pass4SymmKey =
replication_factor = 3

And in the logs for the indexer cluster master I can see this:

09-15-2020 12:56:34.296 +0000 INFO CMMaster - Creating CMMaster: ht=60.000 rf=3 sf=3 ct=60.000 st=60.000 rt=60.000 rct=60.000 rst=60.000 rrt=60.000 rmst=180.000 rmrt=180.000 icps=-1 sfrt=600.000 pe=1 im=1 is=0 mob=2 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=true allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
09-15-2020 12:56:34.296 +0000 WARN CMMaster - pass4SymmKey setting in the clustering or general stanza of server.conf is set to empty or the default value. You must change it to a different value.

Note that server.conf is not totally replaced just the clustering stanzas. So that suggests ansible, but I can't find a anything that changes these stanzas. Note that the search heads are not changed and server.conf is unchanged after the "docker stop". 

 

 

 

Labels (1)
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!