- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am in process of clustering my search heads, but I am receiving the following prompt in the web interface when I click Settings->Search head clustering.
Please wait, the status of your search head cluster is not ready.
Service ready flag: false
Rolling restart in progress: false
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@vulnfree
You've to provide more details about this issue. Check splunkd logs and mongod.log
There are different scenarious which could cause this type of issues:
- KVstore might not be working properly
- Check SSL certificates
- Check your server resources(make sure some times too many data models accelerations could cause this type of issue so try to disable unnecessary data models)
- Are you seeing any skipped searches on search heads?
If this is a non production environment try to follow the below steps and if it is a production please be careful before running the commands . You should have to know what you're doing .
- Stop all Search Head Cluster members and copy the kvstore cd $SPLUNK_HOME/var/lib/splunk/kvstore tar cvfz kvstore-.tar.gz move to a safe place.
- Clean raft and mongod folders
$SPLUNK_HOME/bin/splunk clean kvstore --cluster
$SPLUNK_HOME/bin/splunk clean raft
- Verify all members have replication_factor = 3
$SPLUNK_HOME/bin/splunk btool server list shclustering | grep replication_factor
Start all members
$SPLUNK_HOME/bin/splunk start
Initialize all members
$SPLUNK_HOME/bin/splunk init shcluster-config -auth admin:changed -mgmt_uri https://sh1.example.com:8089 -replication_port 1234 -replication_factor 3 -conf_deploy_fetch_url https://:8089 -secret mykey -shcluster_label shc01
- Verify KVstore
$SPLUNK_HOME/bin/splunk show kvstore-status
- Resync stale KV store members (https://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/ResyncKVstore) > $SPLUNK_HOME/bin/splunk resync kvstore
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@muralikoppula
Do I add the deployment server to the "splunk bootstrap shcluster-captain -servers_list" command?
https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/SHCdeploymentoverview
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@vulnfree
You've to provide more details about this issue. Check splunkd logs and mongod.log
There are different scenarious which could cause this type of issues:
- KVstore might not be working properly
- Check SSL certificates
- Check your server resources(make sure some times too many data models accelerations could cause this type of issue so try to disable unnecessary data models)
- Are you seeing any skipped searches on search heads?
If this is a non production environment try to follow the below steps and if it is a production please be careful before running the commands . You should have to know what you're doing .
- Stop all Search Head Cluster members and copy the kvstore cd $SPLUNK_HOME/var/lib/splunk/kvstore tar cvfz kvstore-.tar.gz move to a safe place.
- Clean raft and mongod folders
$SPLUNK_HOME/bin/splunk clean kvstore --cluster
$SPLUNK_HOME/bin/splunk clean raft
- Verify all members have replication_factor = 3
$SPLUNK_HOME/bin/splunk btool server list shclustering | grep replication_factor
Start all members
$SPLUNK_HOME/bin/splunk start
Initialize all members
$SPLUNK_HOME/bin/splunk init shcluster-config -auth admin:changed -mgmt_uri https://sh1.example.com:8089 -replication_port 1234 -replication_factor 3 -conf_deploy_fetch_url https://:8089 -secret mykey -shcluster_label shc01
- Verify KVstore
$SPLUNK_HOME/bin/splunk show kvstore-status
- Resync stale KV store members (https://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/ResyncKVstore) > $SPLUNK_HOME/bin/splunk resync kvstore
