Search head cluster running 6.3 and Splunk App for Windows Infrastructure 1.20. I'm getting these errors for my scheduled searches:
ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tSessions_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tSessions' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tHostInfo_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tHostInfo' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executedERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;DomainSelector_Lookup", message="Error in 'outputlookup' command: External command based lookup 'DomainSelector' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
On all three search heads, the permissions for /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key is
-rw------- 1 splunk splunk 88 Oct 22 11:42 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
and these errors occur even after splunk is restarted in the environment, so I think that rules out a mongod restart.
This stops the majority of the splunk app for windows infrastructure from displaying results.
In our case, the issue was as a result of the expired certs:
Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator
Here is the fix:
Step 5 is Renamed current server.pem to server.pem.old: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.old
In my case it was that the certificates and other files were too permissive. I did this to fix it:
cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart
Have a look at these two Answer posts:
Mongod fails to start due to SSL cert expiration
Mongod fails to start
It may also be a permissions problem in the mongod tree, so verify that as well.