I had a very similar situation andI realized that some collections were HUGE (in the range of 100 GB), this may cause the mongodb to start very slowly.
I searched in mongodb.log for errors, especially when mongodb starts.
There was not much in there except for some problems while trying to update mongodb to the new version.
I believe that due to its huge size, the service takes too long to starts and goes in conflict with its updates or splunk itself and at the end splunk starts anyway without having the KVstore running.
This is what worked for me, CAREFUL the data will be DELETED from the kvstore, see point [1] if you want to backup the data, but since you are not using it you can just do the clean:
1) Stop the search head that has the stale KV store member.
2) Run the command splunk clean kvstore --local.
3) Restart the search head.
4) Run the command splunk show kvstore-status to verify.
see
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/ResyncKVstore
[1] If you have important data and you dont want to lose it, do a backup and restore
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/BackupKVstore
I hope this helps
... View more