KV store initiation failure-what log is the most relevant for this kind of error?

[Quantum]

KV  store initiation failure, I have got this area that says     ......"error in input lookup command external command-based lookup es notable events is not available because KV store initialization has failed contact your system administrator,"

what log is the most relevant for this kind of error would it be the mongodb log and look for a lock? is that a good route to go?

I am a pretty good engineer but new to Splunk and definitely could use some guidance on just about everything Splunk related.

[Quantum]

05-30-2023 16:58:19.978 -0400 ERROR ExecProcessor [4245 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" Exception in thread "main" java.lang.ExceptionInInitializerError 05-30-2023 16:58:19.978 -0400 ERROR ExecProcessor [4245 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" at com.splunk.dbx.splunkclient.SplunkServiceBuilder.<clinit>(SplunkServiceBuilder.java:19)

/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh"

[Quantum]

2021-05-29T18:15:03.594Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided 2021-05-29T18:15:03.594Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

Thanks for the reply what do you think is this the problem it seems to be that the Mongo database that is the key Value Store does not have a valid certificate and cannot access the application.

[isoutamo]

Quite probably this was the reason. I suppose that you have already found how to fix it? If you are using Splunk's own certs then this describes how to fix it https://community.splunk.com/t5/Security/How-do-I-renew-an-expired-Splunk-Certificate/m-p/389701. If you have own / public certs then do renew process as normally.

If you have changed to another CA than Splunk, then check from conf files that CA etc. files are pointed to correct places. There are couple of settings and some use different attributes for same thing.

[Quantum]

Yeah it looks like it might be expired certificate I am getting that definitely in the Mongo log

[isoutamo]

Hi

you should look both mongodb.log and splunkd.log. You could search those from _internal index with sourcetype mongod or splunkd with host=<your host> source=*/<log file name>. Or look those from file /opt/splunk/var/log/splunk/….
r. Ismo