All Apps and Add-ons
Highlighted

External command based lookup 'tSessions' is not available because KV Store initialization has failed.

Communicator

Search head cluster running 6.3 and Splunk App for Windows Infrastructure 1.20. I'm getting these errors for my scheduled searches:

ERROR SavedSplunker - savedsearchid="nobody;splunkappwindowsinfrastructure;tSessionsLookupUpdate", message="Error in 'inputlookup' command: External command based lookup 'tSessions' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
ERROR SavedSplunker - savedsearchid="nobody;splunkappwindowsinfrastructure;tHostInfoLookupUpdate", message="Error in 'inputlookup' command: External command based lookup 'tHostInfo' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed

ERROR SavedSplunker - savedsearchid="nobody;splunkappwindowsinfrastructure;DomainSelector_Lookup", message="Error in 'outputlookup' command: External command based lookup 'DomainSelector' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
On all three search heads, the permissions for /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key is

-rw------- 1 splunk splunk 88 Oct 22 11:42 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

and these errors occur even after splunk is restarted in the environment, so I think that rules out a mongod restart.
This stops the majority of the splunk app for windows infrastructure from displaying results.

Highlighted

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

Contributor

Have a look at these two Answer posts:
Mongod fails to start due to SSL cert expiration
Mongod fails to start

It may also be a permissions problem in the mongod tree, so verify that as well.

0 Karma
Highlighted

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

Esteemed Legend

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart
0 Karma
Highlighted

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

Explorer

In our case, the issue was as a result of the expired certs:

Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator

Here is the fix:

  1. splunkd.log: ERROR KVStoreLookup - External command based lookup 'app_name' is not available because KV Store initialization has not completed yet......
  2. mongod.log: The provided SSL certificate is expired or not yet valid....
  3. Check Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Sep 8 17:56:51 2019 GMT
  4. Stopped Splunk service
  5. Renamed current server.pem: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem
  6. Restarted Splunk services
  7. Checked Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Oct 24 17:38:28 2022 GMT
  8. Results where displayed for app.
0 Karma
Highlighted

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

Explorer

Step 5 is Renamed current server.pem to server.pem.old: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.old

0 Karma