Knowledge Management

After upgrading to 6.5.0, KV Store will not start

jcrabb_splunk
Splunk Employee
Splunk Employee

After upgrading to Splunk Enterprise 6.5.0, the KV Store will not start. On my indexers I see:

10/5/2016, 5:44:56 AM:

Search peer indexer01.domain.local has the following message: Failed to start KV Store process. See mongod.log and splunkd.log for details.

In splunkd.log I find:

10-05-2016 05:44:56.087 +0000 ERROR MongodRunner - mongod exited abnormally (exit code 14, status: exited with code 14) - look at mongod.log to investigate.

Looking in the mongod.log I find:

2016-10-05T05:44:56.753Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
 2016-10-05T05:44:56.761Z F NETWORK  The provided SSL certificate is expired or not yet valid.
 2016-10-05T05:44:56.761Z I -        Fatal Assertion 28652
 2016-10-05T05:44:56.761Z I -
 ***aborting after fassert() failure

How can this be resolved?

Jacob
Sr. Technical Support Engineer
1 Solution

jcrabb_splunk
Splunk Employee
Splunk Employee

This can happen if the cert used by Splunkd to talk to Mongod has expired. Verify your certs are valid. For example, to validate the expiration date for server.pem you can run:

From $SPLUNK_HOME/etc/auth/

openssl x509 -enddate -noout -in ./server.pem

Results:

notAfter=Dec  10 14:017:25 2015 GMT

In the example above, the cert is expired. If you want to create a new cert you can look at splunk createssl:

$SPLUNK_HOME/bin/splunk help createssl

An example:

$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048

Simply adjust for your environment requirements/settings. Once the new cert is in place, you can test to confirm it is valid:

From $SPLUNK_HOME/etc/auth/

openssl x509 -enddate -noout -in ./server.pem

Results:

notAfter=Aug 22 15:30:45 2019 GMT

If it is now valid, restart Splunk and validate if KVStore is running:

ps -ax | grep mongod

26108 ?        Ssl   62:11 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=50D25A40-7DD2-4017-A223-732705AD4A96 --sslAllowInvalidHostnames --sslMode=preferSSL --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --nounixsocket

And also:

$SPLUNK_HOME/bin/splunk _internal call /services/server/info |grep -i kvstore

<s:key name="kvStoreStatus">ready</s:key>

A third way to check is to run the following search from command line on the instance where you have the "Deployment Management Console / Monitor Console" Configured.

$SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"

          splunk_server       kvStoreStatus
    ------------------------- -------------
    indexer01.domain.com      ready
    indexer02.domain.com      ready
    indexer03.domain.com      ready
    indexer04.domain.com      ready
    indexer05.domain.com      ready
Jacob
Sr. Technical Support Engineer

View solution in original post

benwilinski
New Member

This is the fix. Thank you.

0 Karma

ShaunBaker
Path Finder

Thank you this worked!!

0 Karma

splunksurameric
Engager

Worked for me on Windows 2012. Thank you very much!

0 Karma

julieeball
New Member

Thank you! This worked for me.

0 Karma

a212830
Champion

This is very helpful. I've never had to deal with setting any keys, so I have no idea what our cn.domain.com should be. I'm sure that it's probably right in front of me. Can someone point me in the right direction?

0 Karma

a212830
Champion

Never mind. Duh.

0 Karma

ridwanahmed
Path Finder

I'll ask the question that was in your head originally-- (I've always regenerated by using the method below)-- what IS "cn" in that abstraction?

0 Karma

david_casey
Path Finder

After upgrading to 6.5 I am getting:

1) "Failed to start KV Store process. See mongod.log and splunkd.log for details."
2) "KV Store changed status to failed. KVStore process terminated."
3) "KV Store process terminated abnormally (exit code 1, status exited with code 1)"

I see nothing in Answers related to this. Help!!

0 Karma

david_casey
Path Finder

This has been resolved. Somehow during the upgrade to 6.5 the permissions on splunk.key changed. The permissions on the splunk.key file had to be reset as read-only and the search head rebooted. Problem solved. Shout out to Mike Cormier @ Concanon for his assistance!

Update: Typo fixed - splunk.key is the correct file name.

0 Karma

abhishekkalokhe
Explorer

How to Do this on Windows ?

0 Karma

ChrisBell04
Communicator

My cliff notes for Windows:
open admin cmd prompt

set OPENSSL_CONF=D:\Splunk\openssl.cnf
D:\Splunk\etc\auth>d:\splunk\bin\splunk createssl server-cert -d . -n server
0 Karma

wcgage
Path Finder

I am on a MAC and I don't see the kvstore.key file. Where should it be?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

On macOS nodes it is on /Applications/Splunk/var/lib/splunk/kvstore/mongo/splunk.key. Default seems to be rw instead of r and still it working.

0 Karma

david_casey
Path Finder

/opt/splunk/var/lib/kvstore/mongo/splunk.key

🙂

ndoshi
Splunk Employee
Splunk Employee

To be precise, the file is /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key. Doing a chmod 400 splunk.key did the trick.

0 Karma

damode
Motivator

how to do this on Windows ?
I have Splunk on Windows and I checked the file permission, its already set to full control, still, I am getting the KVstore errors.
PLease help

GArienti
Explorer

can we watch this and get alerted when it's expiring.
mine expired 2 years ago and until I upgraded to 6.5 it didn't come up...

Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...