Knowledge Management

Splunk KV store does not start

rbal_splunk
Splunk Employee
Splunk Employee

KV Store won't start:
I search Splunkbase and folow recommendations to stop splunk, delete mongo.lock and start Unfortunately KV Sore won't start
The $SPLUNK_HOME/var/log/splunk/mongod.log


2015-05-15T20:52:32.253Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:00:39.667Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-05-15T21:00:39.679Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter


Tags (1)

claudio_manig
Communicator

The permissions itself didn't helped out alone in my case - I had another message:
"Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty."
I had to delete the mongod.lock file under $SPLUNKHOME/var/lib/splunk/kvstore/mongo/ and run the mongod --repair command to as well. Restarted splunk and it worked fine.

highsplunker
Contributor

Ok. Nice. This worked for me. But I simply removed this lock-file and restarted Splunk application (if I remember correctly).

0 Karma

rsechser
Engager

Removing the .lock file and restarting the service also fixed it for me. In my case Splunk is on Windows servers.

0 Karma

rgsurfs
Path Finder

I have since transitioned everything to Linux. No issues anymore.

mikaelbje
Motivator

On our servers the Old Splunk certs expired Oct 1 2015. We just upgraded to 6.3 and new certs were not generated. Recreating all certs with ./splunk createssl fixed it.

samaikins
New Member

this is sorted. i removed the old certs and restarted it and that fixed it

0 Karma

anewell
Path Finder

We have a long-serving, much-upgraded server too. The trick that did it for me was to regenerate the certs with the following command:

cd /opt/splunk/etc/auth && /opt/splunk/bin/splunk createssl server-cert -d . -n server 

edjowett
New Member

That was the trick. Thanks.

0 Karma

laurie_gellatly
Communicator

Yep - did the trick for me too. Thanks.

0 Karma

sylax
Explorer

this worked for me also. Make sure you back up the auth directory before making any change

0 Karma

josefa
Path Finder

This worked for me, thanks!

0 Karma

ktweiss
New Member

This worked for me also... quick & simple.
Thank You!

0 Karma

samaikins
New Member

Hi Mikaelbje,
do you mind sharing the steps you took to recreate the ssl certificate

0 Karma

mikaelbje
Motivator

I don't remember exactly, but I think I recreated the web-cert and the server cert. Have a look at the syntax:

$SPLUNK_HOME/bin/splunk help createssl

Back up your old certs first!

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

This turned out to be the permission issue.
splunk runs under splunk account and permission on file was changed to 775 due to recursive permission chnage on $SPLUNK_HOME

/apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key

To resolve this issue we changed the permission back 400.

Chubbybunny
Splunk Employee
Splunk Employee

Thank you! that solved the issue.

0 Karma

DUThibault
Contributor
# chmod og-rwx $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
0 Karma

thambisetty
Super Champion

@DUThibault,

that worked thanks.

————————————
If this helps, give a like below.
0 Karma

rgsurfs
Path Finder

Did that permission change also fix this error:

2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter

????

I have this error and can't clear it.

mattymo
Splunk Employee
Splunk Employee

+1 for cleaning up the perms on the splunk.key on a 6.5 indexer.

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.