topic Re: KV Store status failed in Splunk Enterprise

Why is KV Store status failing?

gl_splunkuser — Fri, 09 Dec 2022 19:39:19 GMT

Hello everyone

I have a situation with the KV Store, from the SH cluster nodes I am getting the next message

KV Store changed status to failed. An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling ismaster on 'servername:8191'

Mongod.log

ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to servername:8191 due to failed operation on a connection

REPL_HB [replexec-3] Error in heartbeat (requestId: 6289) to servername:8191, response status: HostUnreachable: No connection could be made because the target machine actively refused it.
2021-07-01T20:35:18.370Z I

NETWORK [listener] connection accepted from IP:53020 #1194 (12 connections now open)

There is not issues related with the port because the port 8191 is open and I already update certificate of the server.

I have three SH and any of them have the KV status ready.

Do you any idea what can be happening?

Regards.

Re: KV Store status failed

Nemannnnn — Thu, 08 Jul 2021 14:46:59 GMT

Hi how are you? Could you detail me the version of Splunk you use and the operating system installed on each server?

Re: KV Store status failed

gl_splunkuser — Mon, 19 Jul 2021 21:16:47 GMT

The Version:8.0.7 and the clusted was deployed on Windows Server 2016.

Re: KV Store status failed

jessieb_83 — Fri, 17 Jun 2022 21:52:32 GMT

Did you find the solution? Different version, but I'm having the exact same problem.

Re: KV Store status failed

syazwani — Fri, 09 Dec 2022 06:14:54 GMT

Is there any workaround for this issue? Currently im also facing the same issue. Please help.

Re: KV Store status failed

jessieb_83 — Fri, 09 Dec 2022 22:47:15 GMT

I don't remember at this point exactly what I did, but this is the article I used to get through it.

Resync the KV store - Splunk Documentation

Re: KV Store status failed

syazwani — Tue, 13 Dec 2022 01:58:44 GMT

Hi,

Somehow, resync the KV store doesnt work for me. Ive reached to Splunk support and we had resolved the issue. What we did was:

1. regenerating server.pem file

-stop splunk service
- go to cd /opt/splunk/etc/auth and rename the server.pem file to server.pemck
-start splunk

2. clean the kv store locally (on the problematic instance)

- ./splunk stop
- ./splunk clean kvstore --local
-./splunk start

Hope this helps. Thanks.

Re: KV Store status failed

aklon — Mon, 04 Sep 2023 07:45:57 GMT

Below steps fixed the issue

Stop the search head that has the stale KV store member.
Run the command splunk clean kvstore --local.
Restart the search head. This triggers the initial synchronization from other KV store members.
Run the command splunk show kvstore-status to verify synchronization.

Re: KV Store status failed

computermathguy — Tue, 23 Jul 2024 13:20:19 GMT

Worked perfectly. Thank you for posting.