messages shows the below:
Search head cluster member A is having problems pulling configurations from the search head cluster captain B. Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.
any idea regarding the resync commands ??
A Google search for "destructive configuration resync" finds this in Docs: https://docs.splunk.com/Documentation/Splunk/9.2.1/DistSearch/HowconfrepoworksinSHC#Why_a_recovering...
This is the command if using Linux - /opt/splunk/bin/splunk resync shcluster-replicated-config (Run this on other SHC members – not captain)
Check the status first, runs on one of the SHC members
/opt/splunk/bin/splunk show shcluster-status
But might be worth trying a rolling restart and see if that helps, looks like the /var/run folder on the captain is having some kind of issue. Check Disk Space as well. du -sh /opt/splunk/var/run to get the size of the folder.
Rolling restart command = /opt/splunk/bin/splunk rolling-restart shcluster-members from one of the members
Monitor the rolling restart /opt/splunk/bin/splunk rolling-restart shcluster-members -status 1
The captain may change – so observe – you can change the captain back to original
/opt/splunk/bin/splunk transfer shcluster-captain -mgmt_uri <your SHC Captain>
If this is production, then factor in maintenance window or do it when least busy for users, as it will be somewhat disruptive for searches, until it's resolved.
But it It's worth going through the previous answers for this issue, the commands and various steps. Ensure you have backups and plan the steps for the procedure. At minimum /opt/splunk/etc which contains the configuration.