topic Re: Splunk KV store does not start in Knowledge Management

Splunk KV store does not start

rbal_splunk — Sat, 16 May 2015 00:42:49 GMT

KV Store won't start:
I search Splunkbase and folow recommendations to stop splunk, delete mongo.lock and start Unfortunately KV Sore won't start
The $SPLUNK_HOME/var/log/splunk/mongod.log

2015-05-15T20:52:32.253Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:00:39.667Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-05-15T21:00:39.679Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

Re: Splunk KV store does not start

rbal_splunk — Sat, 16 May 2015 00:45:04 GMT

This turned out to be the permission issue.
splunk runs under splunk account and permission on file was changed to 775 due to recursive permission chnage on $SPLUNK_HOME

/apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key

To resolve this issue we changed the permission back 400.

Re: Splunk KV store does not start

Chubbybunny — Mon, 18 May 2015 02:55:28 GMT

Thank you! that solved the issue.

Re: Splunk KV store does not start

rgsurfs — Mon, 19 Oct 2015 14:53:05 GMT

Did that permission change also fix this error:

2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

????

I have this error and can't clear it.

Re: Splunk KV store does not start

mikaelbje — Tue, 20 Oct 2015 14:35:47 GMT

On our servers the Old Splunk certs expired Oct 1 2015. We just upgraded to 6.3 and new certs were not generated. Recreating all certs with ./splunk createssl fixed it.

Re: Splunk KV store does not start

samaikins — Fri, 11 Dec 2015 07:42:22 GMT

Hi Mikaelbje,
do you mind sharing the steps you took to recreate the ssl certificate

Re: Splunk KV store does not start

mikaelbje — Fri, 11 Dec 2015 08:28:52 GMT

I don't remember exactly, but I think I recreated the web-cert and the server cert. Have a look at the syntax:

$SPLUNK_HOME/bin/splunk help createssl

Back up your old certs first!

Re: Splunk KV store does not start

samaikins — Tue, 29 Dec 2015 10:06:23 GMT

this is sorted. i removed the old certs and restarted it and that fixed it

Re: Splunk KV store does not start

anewell — Fri, 05 Feb 2016 23:56:38 GMT

We have a long-serving, much-upgraded server too. The trick that did it for me was to regenerate the certs with the following command:

cd /opt/splunk/etc/auth && /opt/splunk/bin/splunk createssl server-cert -d . -n server

Re: Splunk KV store does not start

josefa — Mon, 29 Aug 2016 21:53:53 GMT

This worked for me, thanks!

Re: Splunk KV store does not start

rgsurfs — Tue, 30 Aug 2016 21:36:58 GMT

I have since transitioned everything to Linux. No issues anymore.

Re: Splunk KV store does not start

claudio_manig — Fri, 11 Nov 2016 12:04:18 GMT

The permissions itself didn't helped out alone in my case - I had another message:
"Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty."
I had to delete the mongod.lock file under $SPLUNKHOME/var/lib/splunk/kvstore/mongo/ and run the mongod --repair command to as well. Restarted splunk and it worked fine.

Re: Splunk KV store does not start

mattymo — Sat, 26 Nov 2016 16:06:59 GMT

+1 for cleaning up the perms on the splunk.key on a 6.5 indexer.

Re: Splunk KV store does not start

DUThibault — Thu, 31 Aug 2017 17:12:46 GMT

# chmod og-rwx $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Re: Splunk KV store does not start

ktweiss — Fri, 19 Jan 2018 14:56:33 GMT

This worked for me also... quick & simple.
Thank You!

Re: Splunk KV store does not start

sylax — Mon, 19 Mar 2018 18:49:45 GMT

this worked for me also. Make sure you back up the auth directory before making any change

Re: Splunk KV store does not start

laurie_gellatly — Fri, 13 Jul 2018 00:35:13 GMT

Yep - did the trick for me too. Thanks.

Re: Splunk KV store does not start

edjowett — Tue, 16 Oct 2018 19:01:58 GMT

That was the trick. Thanks.

Re: Splunk KV store does not start

highsplunker — Thu, 04 Apr 2019 08:46:33 GMT

Ok. Nice. This worked for me. But I simply removed this lock-file and restarted Splunk application (if I remember correctly).

Re: Splunk KV store does not start

rsechser — Thu, 02 May 2019 20:14:32 GMT

Removing the .lock file and restarting the service also fixed it for me. In my case Splunk is on Windows servers.