In Search Head Clustering, what splunkd.log entries will show when an instance may have been the captain or member?

[Ellen]

I know I can run the following to get the current SHC captain,

splunk show shcluster-status -auth <username>:<password>

but for debugging, what text can I search for to see the sequence of the dynamic captain change over time in my SHC instances?

[Ellen · Accepted solution]

In splunkd.log

To verify if the SHC instance is functioning as a captain or member during a timeline, look for these entries:

    Captain:

    10-05-2015 00:13:06.524 +0000 INFO  SHPRaftConsensus - Now leader for term 16
    10-05-2015 00:13:06.524 +0000 INFO  SHPoolingMgr - Making node the captain
    10-05-2015 00:13:06.526 +0000 INFO  SHPoolingMgr - makeOrChangeSlave - master_shp = https://splunk-search-head-cluster-node20:8089

    Member :

    10-05-2015 00:13:06.600 +0000 INFO  SHPRaftConsensus - All hail leader https://splunk-search-head-cluster-node20:8089 for term 16
    10-05-2015 00:13:06.708 +0000 INFO  SHPoolingMgr - makeOrChangeSlave - master_shp = https://splunk-search-head-cluster-node20:8089

[hexx]

I would also suggest to use the Search-Head Clustering views of the Distributed Management Console to this effect. Notably, the "General Status & Configuration" view shows a history of recent captain elections.