Why is KV Store initialization failing on one of our add-on to receive logs?

[khusain_splunk]

While setting up one of our add-on to receive logs, we encountered an issue. While reviewing the internal log we found an error (HTTPError: HTTP 503 error Service Unavailable -- KV store initialization failed . This error also shows up every time splunk services are restarted.

[khusain_splunk · Accepted solution]

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

[tsondo]

A late follow up to this. Updating the certificate made no difference. I am using a third party certificate and it is current and valid. To "fix" it, I backed up my splunk configs, deleted the drive, reinstalled, and put the configs back again. Now it works. Something about the 8 to 9 upgrade just doesn't work. Reinstalling is easier than finding out what went wrong. Fortunately I only have to go through that once. The 9.x updates have not given me any further trouble.

[tsondo]

I am having the same issue, with kv store failing to initialize after upgrade from 8.25 to 9.03. I already copied the correct certificates back to etc/auth, and restarted splunkd, but same issue. Which conf file points Splunk to the correct certificate? Maybe it got replaced in the upgrade and I need to edit it?

[dodland]

Saved my bacon on a Friday afternoon, thank you!!!!

[splunkreal]

Hello,

is this documented in official Splunk docs?

Thanks.

[Mesa_Splunkr]

I am having issues setting up a proofpoint TAP app, here is what the log says.

 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Proofpoint-TAP/bin/proofpoint_tap_siem.py" proofpoint_tap_siem://TAP API: stream_events/HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

I found this article very helpful; however, my certificate is valid, and does not expire till 7/23/2023. My mongod.log also has the following in it.

W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

I am checking the date via GUI when I login to the splunk server. I will research more, wanted to post this to see if you can help. Thanks in advance.

[kcooper]

I just replaced my certificate and the data from our Azure accounts started ingesting again but then it stopped again. 

Received same error:  HTTP 503 Service Unavailable -- KV Store initialization failed.

Any idea how to fix this issue if the certificate is still active? 

[_smp_]

This just saved my a$$. Thanks!

[ssuluguri]

I was getting same error, but after splunk restarted data started collecting