Knowledge Management

Why is KV Store initialization failing on one of our add-on to receive logs?

khusain_splunk
Splunk Employee
Splunk Employee

While setting up one of our add-on to receive logs, we encountered an issue. While reviewing the internal log we found an error (HTTPError: HTTP 503 error Service Unavailable -- KV store initialization failed . This error also shows up every time splunk services are restarted.

0 Karma
1 Solution

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

View solution in original post

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

tsondo
Explorer

A late follow up to this. Updating the certificate made no difference. I am using a third party certificate and it is current and valid. To "fix" it, I backed up my splunk configs, deleted the drive, reinstalled, and put the configs back again. Now it works. Something about the 8 to 9 upgrade just doesn't work. Reinstalling is easier than finding out what went wrong. Fortunately I only have to go through that once. The 9.x updates have not given me any further trouble.

0 Karma

tsondo
Explorer

I am having the same issue, with kv store failing to initialize after upgrade from 8.25 to 9.03. I already copied the correct certificates back to etc/auth, and restarted splunkd, but same issue. Which conf file points Splunk to the correct certificate? Maybe it got replaced in the upgrade and I need to edit it?

dodland
Engager

Saved my bacon on a Friday afternoon, thank you!!!!

0 Karma

splunkreal
Motivator

Hello,

is this documented in official Splunk docs?

Thanks.

 

* If this helps, please upvote or accept solution 🙂 *
0 Karma

Mesa_Splunkr
Loves-to-Learn

I am having issues setting up a proofpoint TAP app, here is what the log says.

 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Proofpoint-TAP/bin/proofpoint_tap_siem.py" proofpoint_tap_siem://TAP API: stream_events/HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

I found this article very helpful; however, my certificate is valid, and does not expire till 7/23/2023. My mongod.log also has the following in it.

W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

I am checking the date via GUI when I login to the splunk server. I will research more, wanted to post this to see if you can help. Thanks in advance.

 

 

0 Karma

kcooper
Communicator

I just replaced my certificate and the data from our Azure accounts started ingesting again but then it stopped again. 

Received same error:  HTTP 503 Service Unavailable -- KV Store initialization failed.

Any idea how to fix this issue if the certificate is still active? 

0 Karma

_smp_
Builder

This just saved my a$$. Thanks!

ssuluguri
Path Finder

I was getting same error, but after splunk restarted data started collecting

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...