Hi
Recently I upgraded my Splunk environment from 6.3.2 to 6.5.0. The environment has a search head cluster, an indexer cluster, and the manager server. After the upgrade, when I run the splunk show shcluster-status
, I noticed there is a new status called last_conf_replication and it always shows Pending for the search head members, but the search head captain doesn't have this. The replication is running fine in the Indexer cluster, so I am not quite sure what this new option is about and how to get the replication finished. Can anyone shed some lights on this?
Thank you.
Hi markusspitzli2
Did Splunk Support advise in which release this bug will be fixed? Splunk 6.6 is recently released but when I went through the release note, I couldn't find the bug fix mentioned.
Hi Daniel.
No. The support is still talking to the development team.
Even for me also, other 2 Search Head Cluster member showing the same status.
last_conf_replication : Pending
Any quick suggestion to fixed it.
We had the same state in our 8 member Searchhead cluster with Splunk 6.5.3. After consulting the Splunk Support it turned out, that we had to use the option "mgmt_uri" instead of "server_list".
After switching the options the pending state disappeared and the cluster synched again.
This behaviour is a confirmed bug in 6.5.3. You can use either mgmt_uri or server_list according to the documentation.
you also can use both options, but you will receive this warning message:
"“It is not advised to use servers_list and mgmt_uri at same time”"
The docs describe the new last_conf_replication state field in the shcluster-status command:
"The last_conf_replication field indicates when the member last pulled a set of configurations from the captain"
http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/ViewSHCstatus#Show_cluster_status
So, it makes sense that the master will not have this status field. Not sure why all of the members would be stuck in a "Pending" state after an upgrade though. Maybe there is just a lot of config information to pull down? Are you seeing any other SHC related log messages that might indicate a problem?