Building for the Splunk Platform

Can you help me with the following KV store error: "KV Store changed status to failed. No suitable servers found"

vrmandadi
Builder

I have the below error message popping up every second

KV Store changed status to failed. No suitable servers found: serverSelectionTimeoutMS expired: [connection closed calling ismaster on 'hesplsrhc004.abc.com:8191']

We have 4 search heads in our cluster and the captain is dynamic. This issue keeps on showing even if search head 4 is shut down.

0 Karma

tokio13
Path Finder

I followed this solution and that solved my problem. (if you experience problem with only one SH, you should perform the solution below at the same time on all the SH)

https://community.splunk.com/t5/Knowledge-Management/Kvstore-has-stuck-at-starting-stage-for-all-the...

0 Karma

SinghK
Builder

So kV store was corrupt.

0 Karma

tokio13
Path Finder

How did you solve this? I know this is an old post but hey, I'm trying..

0 Karma

SinghK
Builder

what do you get when you run splunk show kvstore-status command?

0 Karma

tokio13
Path Finder

For 1 of the 3 SH I get this message (not the capitan):

xxxxxxxxxxxxxxxx-sh1:8191
configVersion : -1
hostAndPort : xxxxxxxxxxxxxxxx-sh1:8191
lastHeartbeat : Mon Feb 28 11:46:21 2022
lastHeartbeatRecv : ZERO_TIME
lastHeartbeatRecvSec : 0
lastHeartbeatSec : 1646041581.267
optimeDate : ZERO_TIME
optimeDateSec : 0
pingMs : 1
replicationStatus : Down
uptime : 0

0 Karma

SinghK
Builder

Can you try and telnet 8191 on other servers. if that works port is not blocked by any firewalls but if that fails somewhere in iptables its blocked.

0 Karma

SinghK
Builder

my bad, can you explain how the environment. Is it a single server like standalone SH or is it cluster?

0 Karma

tokio13
Path Finder

I have a 3SH cluster, 3 x IDX

0 Karma

Vijeta
Influencer

@vrmandadi Were you able to find the solution to this?

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...