I have a cluster master up and running, and was able to add a single search head. Now, when I login to splunkweb on the master and look under clustering, I can see the search head, and see it marked as "Up". I had to do quite a bit to get this to work and may have forgot some of the steps.
Now, I am trying to add a second search head, but am having trouble getting it to join the cluster. The master is reachable on all tcp and udp ports from the search head, but I am getting the following error in splunkd.log:
ERROR ClusteringMgr - VerifyMultisiteConfig failed Error=failed method=GET path=/services/cluster/master/info/?output_mode=json master=splunk-master-01.internal:8089 rv=0 gotConnectionError=1 gotUnexpectedStatusCode=0 actual_response_code=502 expected_response_code=2xx status_line="Error connecting: SSL not configured on client" socket_error="SSL not configured on client" remote_error=
My server.conf looks similar to this:
[general]
serverName = splunk-head-03.internal
site = site1
pass4SymmKey = *****
[sslConfig]
sslPassword = *****
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
[clustering]
master_uri = https://splunk-master-01.internal:8089
multisite = true
mode = searchhead
pass4SymmKey = *****
[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
For SSL, I copied the same certs from the /opt/splunk/etc/auth/mycerts/ directory on the working server and made sure they have the same permissions. Also, I reset sslPassword and pass4SymmKey to the clear text versions each time before I restart the server.
So, what are all the requirements for splunk servers to join the cluster in addition to pass4SymmKey being the same everywhere?
