Knowledge Management

Why is KV Store initialization failing on one of our add-on to receive logs?

khusain_splunk
Splunk Employee
Splunk Employee

While setting up one of our add-on to receive logs, we encountered an issue. While reviewing the internal log we found an error (HTTPError: HTTP 503 error Service Unavailable -- KV store initialization failed . This error also shows up every time splunk services are restarted.

0 Karma
1 Solution

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

View solution in original post

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

dodland
Engager

Saved my bacon on a Friday afternoon, thank you!!!!

0 Karma

realsplunk
Motivator

Hello,

is this documented in official Splunk docs?

Thanks.

 

0 Karma

Mesa_Splunkr
Loves-to-Learn

I am having issues setting up a proofpoint TAP app, here is what the log says.

 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Proofpoint-TAP/bin/proofpoint_tap_siem.py" proofpoint_tap_siem://TAP API: stream_events/HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

I found this article very helpful; however, my certificate is valid, and does not expire till 7/23/2023. My mongod.log also has the following in it.

W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

I am checking the date via GUI when I login to the splunk server. I will research more, wanted to post this to see if you can help. Thanks in advance.

 

 

0 Karma

kcooper
Communicator

I just replaced my certificate and the data from our Azure accounts started ingesting again but then it stopped again. 

Received same error:  HTTP 503 Service Unavailable -- KV Store initialization failed.

Any idea how to fix this issue if the certificate is still active? 

0 Karma

_smp_
Builder

This just saved my a$$. Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...