Why is KV Store status failing?

[gl_splunkuser]

Hello everyone

I have a situation with the KV Store, from the SH cluster nodes I am getting the next message

KV Store changed status to failed. An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling ismaster on 'servername:8191'

Mongod.log

ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to servername:8191 due to failed operation on a connection

REPL_HB [replexec-3] Error in heartbeat (requestId: 6289) to servername:8191, response status: HostUnreachable: No connection could be made because the target machine actively refused it.
2021-07-01T20:35:18.370Z I

NETWORK [listener] connection accepted from IP:53020 #1194 (12 connections now open)

There is not issues related with the port because the port 8191 is open and I already update certificate of the server. 

I have three SH and any of them have the KV status ready. 

Do you any idea what can be happening?

Regards.

[syazwani]

Is there any workaround for this issue? Currently im also facing the same issue. Please help.

[jessieb_83]

I don't remember at this point exactly what I did, but this is the article I used to get through it.

 Resync the KV store - Splunk Documentation

[aklon]

Below steps fixed the issue

1. Stop the search head that has the stale KV store member.
2. Run the command splunk clean kvstore --local.
3. Restart the search head. This triggers the initial synchronization from other KV store members.
4. Run the command splunk show kvstore-status to verify synchronization.

[syazwani]

Hi, 

Somehow, resync the KV store doesnt work for me. Ive reached to Splunk support and we had resolved the issue. What we did was:

1. regenerating server.pem file

-stop splunk service
- go to cd /opt/splunk/etc/auth and rename the server.pem file to server.pemck
-start splunk

2. clean the kv store locally  (on the problematic instance)

- ./splunk stop
- ./splunk clean kvstore --local
-./splunk start

Hope this helps. Thanks.

[computermathguy]

Worked perfectly.  Thank you for posting.

[Nemannnnn]

Hi how are you? Could you detail me the version of Splunk you use and the operating system installed on each server?

[jessieb_83]

Did you find the solution? Different version, but I'm having the exact same problem.

[gl_splunkuser]

The Version:8.0.7 and the clusted was deployed on Windows Server 2016.