Knowledge Management

KV Store failing at SHC

guimilare
Communicator

Hi all.

We are running an environment with a SHC.
After upgrading to v6.6.1, the SH started showing the following message:

"KV Store changed status to failed. Failed to establish communication with KVStore. See splunkd.log for details."

Any ideas what may be the problem?

regards,
GMA

1 Solution

kunalmao
Communicator

Exactly the same issue , and we realised that the issue is with mongo and not splunk.
To solve this you have to set the search head cluster as odd number , we had 4 sh cluster members, we reduced them to 3 by running

splunk remove shcluster-member

on any one of the node.
Now stop Splunk on all of them and run below command

./splunk clean kvstore --local

start splunk again and the error should be gone.

For more details on why Mongo requires odd numbers of nodes refer below

https://stackoverflow.com/questions/16150409/why-does-a-mongodb-replica-set-require-an-odd-number-of...

hope this helps

View solution in original post

kunalmao
Communicator

Exactly the same issue , and we realised that the issue is with mongo and not splunk.
To solve this you have to set the search head cluster as odd number , we had 4 sh cluster members, we reduced them to 3 by running

splunk remove shcluster-member

on any one of the node.
Now stop Splunk on all of them and run below command

./splunk clean kvstore --local

start splunk again and the error should be gone.

For more details on why Mongo requires odd numbers of nodes refer below

https://stackoverflow.com/questions/16150409/why-does-a-mongodb-replica-set-require-an-odd-number-of...

hope this helps

Jarohnimo
Builder

I'd be careful using this on Enteprise Apps like ITSI and ES, this can erase all your relevant data. It's important to backup your KV Store and also Enterprise apps routinely. so in this scenario you could clean the store, restore the kv items and then the JSON files backed up from your enteprise app

0 Karma

koshyk
Super Champion

can you also try the mongod.log

0 Karma

oscarminassian
Path Finder

Thanks for the help guys and sorry for the delay in reply. Turns out we forgot to remove a few searchheads from the master node after a migration and upgrade so things were a little "muddled up". Once removed we did a rolling restart of the search head cluster and things seem to have returned to normal.

0 Karma

oscarminassian
Path Finder

Hi guys,

The exact same problem over here on 6.6.1. We have a SH-Cluster setup and I've managed to clear the issue from one search head by restarting the Splunkd service, only to have it move over to another search head in the cluster! Rince repeat, the issue will only ever be on one search head at a time. I haven't observed any down the line impact, but the error is annoying!

Oscar

0 Karma

kunalmao
Communicator

It is getting moved to other search head as the issue resides on the search head captain. Even we are facing same issue.

0 Karma

Azeemering
Builder

What does the splunkd log say in detail?

0 Karma

guimilare
Communicator

06-29-2017 18:00:07.014 +0000 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to establish communication with KVStore. See splunkd.log for details.

06-29-2017 18:06:04.242 +0000 WARN SHPConfReplicationBookmarkFromKVStore - writeSHCStateToStateStore: failed to write SHC state to KVstore:

Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...