Knowledge Management

Why is KV Store initialization failing on one of our add-on to receive logs?

khusain_splunk
Splunk Employee
Splunk Employee

While setting up one of our add-on to receive logs, we encountered an issue. While reviewing the internal log we found an error (HTTPError: HTTP 503 error Service Unavailable -- KV store initialization failed . This error also shows up every time splunk services are restarted.

0 Karma
1 Solution

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

View solution in original post

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

Please check mongod.log under $SPLUNK_HOME/var/log/splunk/, if it says related to SSL certificate, exp:

The provided SSL certificate is expired or not yet valid.
No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile

Then, you need to renew the SSL certificate. If you are using third-party certificate then place the new certificate and restart splunkd. Else, if you are on default certificate, go under $SPLUNK_HOME/etc/auth/ and rename server.pem file and restart the splunk which will generate the new SSL certificate and kv store will be up .

Thanks
Kashif Husain

tsondo
Engager

I am having the same issue, with kv store failing to initialize after upgrade from 8.25 to 9.03. I already copied the correct certificates back to etc/auth, and restarted splunkd, but same issue. Which conf file points Splunk to the correct certificate? Maybe it got replaced in the upgrade and I need to edit it?

dodland
Engager

Saved my bacon on a Friday afternoon, thank you!!!!

0 Karma

realsplunk
Motivator

Hello,

is this documented in official Splunk docs?

Thanks.

 

0 Karma

Mesa_Splunkr
Loves-to-Learn

I am having issues setting up a proofpoint TAP app, here is what the log says.

 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-Proofpoint-TAP/bin/proofpoint_tap_siem.py" proofpoint_tap_siem://TAP API: stream_events/HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

I found this article very helpful; however, my certificate is valid, and does not expire till 7/23/2023. My mongod.log also has the following in it.

W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

I am checking the date via GUI when I login to the splunk server. I will research more, wanted to post this to see if you can help. Thanks in advance.

 

 

0 Karma

kcooper
Communicator

I just replaced my certificate and the data from our Azure accounts started ingesting again but then it stopped again. 

Received same error:  HTTP 503 Service Unavailable -- KV Store initialization failed.

Any idea how to fix this issue if the certificate is still active? 

0 Karma

_smp_
Builder

This just saved my a$$. Thanks!

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...