I am receiving this error on my CM. The peer nodes are not able to join CM
What is preventing them from joining the cluster? Are there network or host based firewalls blocking the traffic? Are the peer nodes properly configured?
We need more information to help.
On the indexer boxes I am receving 1069 error that splunkweb services are not running. You think that could be the issue?
splunkweb doesn't need to be running on the indexers, so that isn't necessarily pointing to the issue. Is splunkd running on the indexers?
yes Splunkd is running on the indexers. But splunk webservice is not.
Are the indexers configured to be a part of the cluster? What do you get for this command when run on an indexer:
splunk btool server list clustering --debug
[general]
serverName = Server-name
pass4SymmKey = $1$tVLM068URVKj
[queue]
maxSize = 0
[queue=indexQueue]
maxSize = 1GB
[queue=parsingQueue]
maxSize = 100MB
[queue=aggQueue]
maxSize = 100MB
[queue=auditQueue]
maxSize = 100MB
[queue=typingQueue]
maxSize = 10GB
[queue=tcpout_my_indexers]
maxSize = 100MB
[sslConfig]
sslPassword = $1$4h6Yj+pSDgCj
allowSslCompression=false
[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[license]
master_uri = https://10.55.3.23:8089
[diskUsage]
minFreeSpace = 500
Looks ok to me
Unless your post has been cropped, you have no cluster config there at all.
You should have a section like:
[clustering]
master_uri = xxxxx:xxx
pass4SymmKey = xxxx
...
...
How did you configure the peers, through the UI, or via the cli with something like
splunk edit cluster-config -mode slave -master_uri https://10.160.31.200:8089 -replication_port 9887 -secret your_key
?
https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/ConfigurepeerswithCLI
search have a look at peer nodes
cat splunkd.log |grep -i error
or provide log
its a windows box