Splunk Enterprise

Why is KV Store status failing?

gl_splunkuser
Path Finder

Hello everyone

I have a situation with the KV Store, from the SH cluster nodes I am getting the next message

KV Store changed status to failed. An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling ismaster on 'servername:8191'

Mongod.log

ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to servername:8191 due to failed operation on a connection

REPL_HB [replexec-3] Error in heartbeat (requestId: 6289) to servername:8191, response status: HostUnreachable: No connection could be made because the target machine actively refused it.
2021-07-01T20:35:18.370Z I

NETWORK [listener] connection accepted from IP:53020 #1194 (12 connections now open)

 

There is not issues related with the port because the port 8191 is open and I already update certificate of the server. 

I have three SH and any of them have the KV status ready. 

Do you any idea what can be happening?

 

Regards.

 

 

Labels (2)
0 Karma

syazwani
Path Finder

Is there any workaround for this issue? Currently im also facing the same issue. Please help.

0 Karma

jessieb_83
Explorer

I don't remember at this point exactly what I did, but this is the article I used to get through it.

 Resync the KV store - Splunk Documentation

aklon
Engager

Below steps fixed the issue

  1. Stop the search head that has the stale KV store member.
  2. Run the command splunk clean kvstore --local.
  3. Restart the search head. This triggers the initial synchronization from other KV store members.
  4. Run the command splunk show kvstore-status to verify synchronization.
Tags (1)
0 Karma

syazwani
Path Finder

Hi, 

Somehow, resync the KV store doesnt work for me. Ive reached to Splunk support and we had resolved the issue. What we did was:

1. regenerating server.pem file

-stop splunk service
- go to cd /opt/splunk/etc/auth and rename the server.pem file to server.pemck
-start splunk

2. clean the kv store locally  (on the problematic instance)

- ./splunk stop
- ./splunk clean kvstore --local
-./splunk start

Hope this helps. Thanks.

0 Karma

Nemannnnn
New Member

Hi how are you? Could you detail me the version of Splunk you use and the operating system installed on each server?

0 Karma

jessieb_83
Explorer

Did you find the solution? Different version, but I'm having the exact same problem.

0 Karma

gl_splunkuser
Path Finder

The Version:8.0.7 and the clusted was deployed on Windows Server 2016.

 

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...