How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"

vulnfree — Thu, 03 Jan 2019 16:10:07 GMT

I am in process of clustering my search heads, but I am receiving the following prompt in the web interface when I click Settings->Search head clustering.

Please wait, the status of your search head cluster is not ready.

Service ready flag: false

Rolling restart in progress: false

Re: How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"

muralikoppula — Tue, 29 Sep 2020 22:34:01 GMT

@vulnfree
You've to provide more details about this issue. Check splunkd logs and mongod.log

There are different scenarious which could cause this type of issues:
- KVstore might not be working properly
- Check SSL certificates
- Check your server resources(make sure some times too many data models accelerations could cause this type of issue so try to disable unnecessary data models)
- Are you seeing any skipped searches on search heads?

If this is a non production environment try to follow the below steps and if it is a production please be careful before running the commands . You should have to know what you're doing .

Stop all Search Head Cluster members and copy the kvstore cd $SPLUNK_HOME/var/lib/splunk/kvstore tar cvfz kvstore-.tar.gz move to a safe place.
Clean raft and mongod folders

$SPLUNK_HOME/bin/splunk clean kvstore --cluster
$SPLUNK_HOME/bin/splunk clean raft

Verify all members have replication_factor = 3

$SPLUNK_HOME/bin/splunk btool server list shclustering | grep replication_factor

Start all members

$SPLUNK_HOME/bin/splunk start
Initialize all members

$SPLUNK_HOME/bin/splunk init shcluster-config -auth admin:changed -mgmt_uri https://sh1.example.com:8089 -replication_port 1234 -replication_factor 3 -conf_deploy_fetch_url https://:8089 -secret mykey -shcluster_label shc01

Verify KVstore

$SPLUNK_HOME/bin/splunk show kvstore-status

Resync stale KV store members (https://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/ResyncKVstore) > $SPLUNK_HOME/bin/splunk resync kvstore

Re: How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"

vulnfree — Fri, 11 Jan 2019 22:22:39 GMT

@muralikoppula

Do I add the deployment server to the "splunk bootstrap shcluster-captain -servers_list" command?

https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/SHCdeploymentoverview

topic Re: How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready" in Deployment Architecture

How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"

Re: How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"

Re: How come I'm receiving the following prompt?: "Search Head Clustering Service Not Ready"