topic Re: KV store initiation failure, in Splunk Enterprise

KV store initiation failure-what log is the most relevant for this kind of error?

Quantum — Wed, 31 May 2023 14:50:49 GMT

KV store initiation failure, I have got this area that says ......"error in input lookup command external command-based lookup es notable events is not available because KV store initialization has failed contact your system administrator,"

what log is the most relevant for this kind of error would it be the mongodb log and look for a lock? is that a good route to go?

I am a pretty good engineer but new to Splunk and definitely could use some guidance on just about everything Splunk related.

Re: KV store initiation failure,

isoutamo — Tue, 30 May 2023 20:51:29 GMT

you should look both mongodb.log and splunkd.log. You could search those from _internal index with sourcetype mongod or splunkd with host=<your host> source=*/<log file name>. Or look those from file /opt/splunk/var/log/splunk/….
r. Ismo

Re: KV store initiation failure,

Quantum — Tue, 30 May 2023 21:29:32 GMT

Yeah it looks like it might be expired certificate I am getting that definitely in the Mongo log

Re: KV store initiation failure,

Quantum — Tue, 30 May 2023 21:32:30 GMT

2021-05-29T18:15:03.594Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided 2021-05-29T18:15:03.594Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

Thanks for the reply what do you think is this the problem it seems to be that the Mongo database that is the key Value Store does not have a valid certificate and cannot access the application.

Re: KV store initiation failure,

Quantum — Tue, 30 May 2023 21:35:19 GMT

05-30-2023 16:58:19.978 -0400 ERROR ExecProcessor [4245 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" Exception in thread "main" java.lang.ExceptionInInitializerError 05-30-2023 16:58:19.978 -0400 ERROR ExecProcessor [4245 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" at com.splunk.dbx.splunkclient.SplunkServiceBuilder.<clinit>(SplunkServiceBuilder.java:19)

/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh"

Re: KV store initiation failure,

isoutamo — Wed, 31 May 2023 06:12:48 GMT

Quite probably this was the reason. I suppose that you have already found how to fix it? If you are using Splunk's own certs then this describes how to fix it https://community.splunk.com/t5/Security/How-do-I-renew-an-expired-Splunk-Certificate/m-p/389701. If you have own / public certs then do renew process as normally.

If you have changed to another CA than Splunk, then check from conf files that CA etc. files are pointed to correct places. There are couple of settings and some use different attributes for same thing.