topic Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed. in All Apps and Add-ons

External command based lookup 'tSessions' is not available because KV Store initialization has failed.

banderson7 — Tue, 29 Sep 2020 08:26:33 GMT

Search head cluster running 6.3 and Splunk App for Windows Infrastructure 1.20. I'm getting these errors for my scheduled searches:

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tSessions_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tSessions' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tHostInfo_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tHostInfo' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;DomainSelector_Lookup", message="Error in 'outputlookup' command: External command based lookup 'DomainSelector' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
On all three search heads, the permissions for /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key is

-rw------- 1 splunk splunk 88 Oct 22 11:42 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

and these errors occur even after splunk is restarted in the environment, so I think that rules out a mongod restart.
This stops the majority of the splunk app for windows infrastructure from displaying results.

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

nnmiller — Fri, 06 May 2016 12:35:19 GMT

Have a look at these two Answer posts:
Mongod fails to start due to SSL cert expiration
Mongod fails to start

It may also be a permissions problem in the mongod tree, so verify that as well.

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

woodcock — Sat, 28 Jul 2018 02:37:32 GMT

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

daymauler — Fri, 25 Oct 2019 17:49:46 GMT

In our case, the issue was as a result of the expired certs:

Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator

Here is the fix:

splunkd.log: ERROR KVStoreLookup - External command based lookup 'app_name' is not available because KV Store initialization has not completed yet......
mongod.log: The provided SSL certificate is expired or not yet valid....
Check Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Sep 8 17:56:51 2019 GMT
Stopped Splunk service
Renamed current server.pem: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem
Restarted Splunk services
Checked Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Oct 24 17:38:28 2022 GMT
Results where displayed for app.

Re: External command based lookup 'tSessions' is not available because KV Store initialization has failed.

daymauler — Mon, 28 Oct 2019 11:29:04 GMT

Step 5 is Renamed current server.pem to server.pem.old: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.old