Deployment Architecture

Splunk SH Cluster - KV store cannot initiate set

perfecto25
Path Finder

Hello, Im running a 3-node SH cluster + deployer (running splunk 7.1 on Centos 7)

Cluster is up and Captain is SH1, my kv-status comes back with,

This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 6DE38BA1-8716-4909-9053-C60751902220
                                          port : 8191
                                    standalone : 0
                                        status : failed

 Enabled KV store members:
        njosplunksh02.company.local:8191
                                          guid : 6C547544-700B-4A12-9446-C2246E73A717
                                   hostAndPort : njosplunksh02.company.local:8191
        njosplunksh01.company.local:8191
                                          guid : 6DE38BA1-8716-4909-9053-C60751902220
                                   hostAndPort : njosplunksh01.company.local:8191
        njosplunksh03.company.local:8191
                                          guid : 85A78216-32F6-4875-8FC7-9DB7BB0AD1E5
                                   hostAndPort : njosplunksh03.company.local:8191

On the 1st SH (captain), Im getting this warning in the console,

KV Store changed status to failed. 'njosplunksh01.company.local:8191' has data already, cannot initiate set.

I tried cleaning Raft and KV Store, but getting the same results after splunk is restarted,

"rm -rf /opt/splunk/var/run/splunk/_raft/*"
"/opt/splunk/bin/splunk clean kvstore --cluster --answer-yes"
"/opt/splunk/bin/splunk clean raft --answer-yes"

Couldnt find anything in KB for this error. Does anyone know how to troubleshoot this? Thanks.

0 Karma

leonardoguerra1
New Member

Hi, could you solve this problem? If you solved it, can you tell me how?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...