Why am I getting KV store error 'External command based lookup 'MY_KV_LOOKUP' is not available because KV Store initialization has failed'?

[pedromvieira]

Hello,

After a while, my KV Store isnt working.

I receive this message:

The lookup table 'External command based lookup 'MY_KV_LOOKUP' is not available because KV Store initialization has failed. Please contact your system administrator.' is invalid.

Nothings shows wrong when I restart Splunk or in _internal logs.

[woodcock]

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart

[giorgio79dis]

Thank you! Your solution worked

[woodcock]

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart

[pedromvieira · Accepted solution]

After manually restarting mongod service, it worked.

[jkellerman_splu]

Hi, I had the exact same behaviour but in my case the root cause was a firewall between the search head cluster peers that did not allow communication on kvstore replication port (default 8191)

[ashnet16]

How did you perform your mongod service restart? Thanks.

[bravon]

For me the solution was (and still is) to deleted the mongod.lock file and do a "Splunk restart"

[pasokkum]

I have tried this option also but th issue has not resolved. Is there any alternative

[cafissimo]

Please, How did you restarted mongod service? Are you running splunk on windows or Linux

Thanks

[kundeng]

I tried to restart mongod on linux, but it requires some ssl lib file under splunk/lib.

Anyone has tried and successfully restarted mongod service?

[cedarcrestone]

I am experiencing the same issue. Any lucks on the command for Linux?

[peteharverson]

I just experienced the same issue - Splunk 6.2.3 running on Linux. As well as deleting mongod.lock, I also had to delete splunk.key, both under $SPLUNK_HOME/var/lib/splunk/kvstore/mongo, before doing a splunk restart.