Hello,
After a while, my KV Store isnt working.
I receive this message:
The lookup table 'External command based lookup 'MY_KV_LOOKUP' is not available because KV Store initialization has failed. Please contact your system administrator.' is invalid.
Nothings shows wrong when I restart Splunk or in _internal logs.
In my case it was that the certificates and other files were too permissive. I did this to fix it:
cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart
Thank you! Your solution worked
In my case it was that the certificates and other files were too permissive. I did this to fix it:
cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart
After manually restarting mongod service, it worked.
Hi, I had the exact same behaviour but in my case the root cause was a firewall between the search head cluster peers that did not allow communication on kvstore replication port (default 8191)
How did you perform your mongod service restart? Thanks.
For me the solution was (and still is) to deleted the mongod.lock file and do a "Splunk restart"
I have tried this option also but th issue has not resolved. Is there any alternative
Please, How did you restarted mongod service? Are you running splunk on windows or Linux
Thanks
I tried to restart mongod on linux, but it requires some ssl lib file under splunk/lib.
Anyone has tried and successfully restarted mongod service?
I am experiencing the same issue. Any lucks on the command for Linux?
I just experienced the same issue - Splunk 6.2.3 running on Linux. As well as deleting mongod.lock, I also had to delete splunk.key, both under $SPLUNK_HOME/var/lib/splunk/kvstore/mongo, before doing a splunk restart.