Can you help me with the following KV store error: "KV Store changed status to failed. No suitable servers found"

[vrmandadi]

I have the below error message popping up every second

KV Store changed status to failed. No suitable servers found: serverSelectionTimeoutMS expired: [connection closed calling ismaster on 'hesplsrhc004.abc.com:8191']

We have 4 search heads in our cluster and the captain is dynamic. This issue keeps on showing even if search head 4 is shut down.

[tokio13]

I followed this solution and that solved my problem. (if you experience problem with only one SH, you should perform the solution below at the same time on all the SH)

https://community.splunk.com/t5/Knowledge-Management/Kvstore-has-stuck-at-starting-stage-for-all-the...

[SinghK]

So kV store was corrupt.

[tokio13]

How did you solve this? I know this is an old post but hey, I'm trying..

[SinghK]

what do you get when you run splunk show kvstore-status command?

[tokio13]

For 1 of the 3 SH I get this message (not the capitan):

xxxxxxxxxxxxxxxx-sh1:8191
configVersion : -1
hostAndPort : xxxxxxxxxxxxxxxx-sh1:8191
lastHeartbeat : Mon Feb 28 11:46:21 2022
lastHeartbeatRecv : ZERO_TIME
lastHeartbeatRecvSec : 0
lastHeartbeatSec : 1646041581.267
optimeDate : ZERO_TIME
optimeDateSec : 0
pingMs : 1
replicationStatus : Down
uptime : 0

[SinghK]

Can you try and telnet 8191 on other servers. if that works port is not blocked by any firewalls but if that fails somewhere in iptables its blocked.

[SinghK]

my bad, can you explain how the environment. Is it a single server like standalone SH or is it cluster?

[tokio13]

I have a 3SH cluster, 3 x IDX

[Vijeta]

@vrmandadi Were you able to find the solution to this?