All Apps and Add-ons

External command based lookup 'tSessions' is not available because KV Store initialization has failed.

banderson7
Communicator

Search head cluster running 6.3 and Splunk App for Windows Infrastructure 1.20. I'm getting these errors for my scheduled searches:

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tSessions_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tSessions' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tHostInfo_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tHostInfo' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;DomainSelector_Lookup", message="Error in 'outputlookup' command: External command based lookup 'DomainSelector' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
On all three search heads, the permissions for /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key is

-rw------- 1 splunk splunk 88 Oct 22 11:42 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

and these errors occur even after splunk is restarted in the environment, so I think that rules out a mongod restart.
This stops the majority of the splunk app for windows infrastructure from displaying results.

daymauler
Explorer

In our case, the issue was as a result of the expired certs:

Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator

Here is the fix:

  1. splunkd.log: ERROR KVStoreLookup - External command based lookup 'app_name' is not available because KV Store initialization has not completed yet......
  2. mongod.log: The provided SSL certificate is expired or not yet valid....
  3. Check Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Sep 8 17:56:51 2019 GMT
  4. Stopped Splunk service
  5. Renamed current server.pem: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem
  6. Restarted Splunk services
  7. Checked Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Oct 24 17:38:28 2022 GMT
  8. Results where displayed for app.
0 Karma

daymauler
Explorer

Step 5 is Renamed current server.pem to server.pem.old: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.old

0 Karma

woodcock
Esteemed Legend

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart
0 Karma

nnmiller
Contributor

Have a look at these two Answer posts:
Mongod fails to start due to SSL cert expiration
Mongod fails to start

It may also be a permissions problem in the mongod tree, so verify that as well.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...