Hi waddellt, please check this article to troubleshoot the kvstore
https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/TroubleshootKVstore
Per the article it seems that your failed kvstore message is related to:
failed - Failed to bootstrap and join the search head cluster.
if you are working on a Splunk Enterprise Security search head cluster you can also run a command to resync or if it did not work, clean-up the kvstore for this particular server.
try first :
- Resync kvstore (https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/ResyncKVstore#Resync_stale_KV_store_members)
- splunk resync kvstore [-source sourceId]
Note: if you are running on a cluster, please manual run a backup on the kvstore from a note that kvstore is working properly, check this procedure here(https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/BackupKVstore)
Or if you are running on a stand alone instance you can clean the kvstore. Please be carefully, because it will reset all the data into the kvstore and you can lose the data that was there. On the previous link I provided, you have the both commands.
splunk clean kvstore --local
