We have tried doing a rolling restart but all members are stuck on their current state.
We are also seeing some errors in the mongod.log file like this:
2016-01-15T02:01:54.263Z E REPL [conn201] replSet replSetInitiate failed; CannotInitializeNodeWithData 'test.machine.com:8191' has data already, cannot initiate set.
2016-01-15T02:01:54.263Z W REPL [ReplicationExecutor] 'test.machine.com:8191' has data already, cannot initiate set.
Verify that EACH member has access to all other members (you can use telnet host port to do that). We have seen several times, when just one member has some wrong routing and because of that KVStore cannot elect the captain. One of the reasons - wrong records in /etc/hosts.
KVStore depends on SHC, if SHC is in bad state - KVStore will not work properly. So first verify that SHC is available, all members are listed and SHC has an elected Captain.
splunk show shcluster-status
After you will find out who is the SHC captain it is good to start investigation from the SHC Captain what is wrong with the KVStore cluster.
The first one (mongod.log) can have information about why mongod could not be started or crash log. The second one (splunkd.log) can have information about why splunk could not launch mongod or why it changed status of KVStore to failed.