Deployment Architecture

Splunk SH Cluster - KV store cannot initiate set

perfecto25
Path Finder

Hello, Im running a 3-node SH cluster + deployer (running splunk 7.1 on Centos 7)

Cluster is up and Captain is SH1, my kv-status comes back with,

This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 6DE38BA1-8716-4909-9053-C60751902220
                                          port : 8191
                                    standalone : 0
                                        status : failed

 Enabled KV store members:
        njosplunksh02.company.local:8191
                                          guid : 6C547544-700B-4A12-9446-C2246E73A717
                                   hostAndPort : njosplunksh02.company.local:8191
        njosplunksh01.company.local:8191
                                          guid : 6DE38BA1-8716-4909-9053-C60751902220
                                   hostAndPort : njosplunksh01.company.local:8191
        njosplunksh03.company.local:8191
                                          guid : 85A78216-32F6-4875-8FC7-9DB7BB0AD1E5
                                   hostAndPort : njosplunksh03.company.local:8191

On the 1st SH (captain), Im getting this warning in the console,

KV Store changed status to failed. 'njosplunksh01.company.local:8191' has data already, cannot initiate set.

I tried cleaning Raft and KV Store, but getting the same results after splunk is restarted,

"rm -rf /opt/splunk/var/run/splunk/_raft/*"
"/opt/splunk/bin/splunk clean kvstore --cluster --answer-yes"
"/opt/splunk/bin/splunk clean raft --answer-yes"

Couldnt find anything in KB for this error. Does anyone know how to troubleshoot this? Thanks.

0 Karma

leonardoguerra1
New Member

Hi, could you solve this problem? If you solved it, can you tell me how?

0 Karma
Get Updates on the Splunk Community!

Announcing the Expansion of the Splunk Academic Alliance Program

The Splunk Community is more than just an online forum — it’s a network of passionate users, administrators, ...

Learn Splunk Insider Insights, Do More With Gen AI, & Find 20+ New Use Cases You Can ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Buttercup Games: Further Dashboarding Techniques (Part 7)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...