KV Store won't start:
I search Splunkbase and folow recommendations to stop splunk, delete mongo.lock and start Unfortunately KV Sore won't start
The $SPLUNK_HOME/var/log/splunk/mongod.log
2015-05-15T20:52:32.253Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:00:39.667Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-05-15T21:00:39.679Z permissions on /apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
The permissions itself didn't helped out alone in my case - I had another message:
"Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty."
I had to delete the mongod.lock file under $SPLUNKHOME/var/lib/splunk/kvstore/mongo/ and run the mongod --repair command to as well. Restarted splunk and it worked fine.
Ok. Nice. This worked for me. But I simply removed this lock-file and restarted Splunk application (if I remember correctly).
Removing the .lock file and restarting the service also fixed it for me. In my case Splunk is on Windows servers.
I have since transitioned everything to Linux. No issues anymore.
On our servers the Old Splunk certs expired Oct 1 2015. We just upgraded to 6.3 and new certs were not generated. Recreating all certs with ./splunk createssl fixed it.
this is sorted. i removed the old certs and restarted it and that fixed it
We have a long-serving, much-upgraded server too. The trick that did it for me was to regenerate the certs with the following command:
cd /opt/splunk/etc/auth && /opt/splunk/bin/splunk createssl server-cert -d . -n server
That was the trick. Thanks.
Yep - did the trick for me too. Thanks.
this worked for me also. Make sure you back up the auth directory before making any change
This worked for me, thanks!
This worked for me also... quick & simple.
Thank You!
Hi Mikaelbje,
do you mind sharing the steps you took to recreate the ssl certificate
I don't remember exactly, but I think I recreated the web-cert and the server cert. Have a look at the syntax:
$SPLUNK_HOME/bin/splunk help createssl
Back up your old certs first!
This turned out to be the permission issue.
splunk runs under splunk account and permission on file was changed to 775 due to recursive permission chnage on $SPLUNK_HOME
/apps/splunk/var/lib/splunk/kvstore/mongo/splunk.key
To resolve this issue we changed the permission back 400.
Thank you! that solved the issue.
# chmod og-rwx $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
@DUThibault,
that worked thanks.
Did that permission change also fix this error:
2015-05-15T21:08:00.418Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
????
I have this error and can't clear it.
+1
for cleaning up the perms on the splunk.key on a 6.5 indexer.