- Splunk Answers
- :
- Using Splunk
- :
- Splunk Dev
- :
- Re: Can you help me with the following KV store er...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you help me with the following KV store error: "KV Store changed status to failed. No suitable servers found"
I have the below error message popping up every second
KV Store changed status to failed. No suitable servers found: serverSelectionTimeoutMS expired: [connection closed calling ismaster on 'hesplsrhc004.abc.com:8191']
We have 4 search heads in our cluster and the captain is dynamic. This issue keeps on showing even if search head 4 is shut down.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I followed this solution and that solved my problem. (if you experience problem with only one SH, you should perform the solution below at the same time on all the SH)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So kV store was corrupt.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How did you solve this? I know this is an old post but hey, I'm trying..
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
what do you get when you run splunk show kvstore-status command?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For 1 of the 3 SH I get this message (not the capitan):
xxxxxxxxxxxxxxxx-sh1:8191
configVersion : -1
hostAndPort : xxxxxxxxxxxxxxxx-sh1:8191
lastHeartbeat : Mon Feb 28 11:46:21 2022
lastHeartbeatRecv : ZERO_TIME
lastHeartbeatRecvSec : 0
lastHeartbeatSec : 1646041581.267
optimeDate : ZERO_TIME
optimeDateSec : 0
pingMs : 1
replicationStatus : Down
uptime : 0
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you try and telnet 8191 on other servers. if that works port is not blocked by any firewalls but if that fails somewhere in iptables its blocked.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
my bad, can you explain how the environment. Is it a single server like standalone SH or is it cluster?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have a 3SH cluster, 3 x IDX
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@vrmandadi Were you able to find the solution to this?
Routing logs with Splunk OTel Collector for Kubernetes
Welcome to the Splunk Community!
Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM
