- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All
I have recently taken over the admin of our splunk server, I upgraded to 7.2.0 and its been running fine for a while, yesterday we started getting errors:
Failed to start KV Store process. See mongod.log and splunkd.log for details.
11/13/2018, 9:09:18 AM
KV Store changed status to failed. KVStore process terminated.
11/13/2018, 9:09:16 AM
KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details.
11/13/2018, 9:09:16 AM
after looking that up I saw that the internal SSL cert had expired so I renewed it as per the instructions:
"set OPENSSL_CONF=D:\Splunk\openssl.cnf
D:\Splunk\etc\auth>d:\splunk\bin\splunk createssl server-cert -d . -n server"
This is now showing the cert to be valid. But now, I am getting the error below in the mongod log file.
2018-11-13T09:09:16.227Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] MongoDB starting : pid=8104 port=8191 dbpath=E:\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=PRDSPLKAPP02
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] db version v3.6.7
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2o-fips 27 Mar 2018
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] allocator: tcmalloc
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] modules: none
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] build environment:
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] distarch: x86_64
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] target_arch: x86_64
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "E:\Splunk\etc\auth\server.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "C3E895A2-5F0A-4968-856E-C1C0047199B9" }, security: { javascriptEnabled: false, keyFile: "E:\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "E:\Splunk\var\lib\splunk\kvstore\mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2018-11-13T09:09:16.401Z I JOURNAL [initandlisten] journal dir=E:\Splunk\var\lib\splunk\kvstore\mongo\journal
2018-11-13T09:09:16.401Z I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
2018-11-13T09:09:16.457Z I JOURNAL [durability] Durability thread started
2018-11-13T09:09:16.458Z I JOURNAL [journal writer] Journal writer thread started
2018-11-13T09:09:16.460Z I CONTROL [initandlisten]
2018-11-13T09:09:16.460Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
2018-11-13T09:09:16.460Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.
2018-11-13T09:09:16.488Z F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
2018-11-13T09:09:16.488Z I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutdown: removing all drop-pending collections...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutdown: removing checkpointTimestamp collection...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutting down replication subsystems
2018-11-13T09:09:16.488Z W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished. Shutting down without cleaning up the replication system
2018-11-13T09:09:16.488Z I STORAGE [initandlisten] shutdown: waiting for fs preallocator...
2018-11-13T09:09:16.488Z I STORAGE [initandlisten] shutdown: final commit...
2018-11-13T09:09:16.492Z I JOURNAL [initandlisten] journalCleanup...
2018-11-13T09:09:16.492Z I JOURNAL [initandlisten] removeJournalFiles
2018-11-13T09:09:16.497Z I JOURNAL [initandlisten] old journal file E:\Splunk\var\lib\splunk\kvstore\mongo\journal\j._0 will be reused as E:\Splunk\var\lib\splunk\kvstore\mongo\journal\prealloc.0
2018-11-13T09:09:16.498Z I JOURNAL [initandlisten] Terminating durability thread ...
2018-11-13T09:09:16.521Z I JOURNAL [journal writer] Journal writer thread stopped
2018-11-13T09:09:16.521Z I JOURNAL [durability] Durability thread stopped
2018-11-13T09:09:16.521Z I STORAGE [initandlisten] shutdown: closing all files...
2018-11-13T09:09:16.534Z I STORAGE [initandlisten] closeAllFiles() finished
2018-11-13T09:09:16.534Z I STORAGE [initandlisten] shutdown: removing fs lock...
2018-11-13T09:09:16.535Z I CONTROL [initandlisten] now exiting
2018-11-13T09:09:16.535Z I CONTROL [initandlisten] shutting down with code:62
The two big errors being:
"Please specify an sslCAFile parameter."
where do I specify this?
** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
would this not have upgraded with the version of Splunk? if not, how do I upgrade this?
any help would be appreciated, Thank you
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
resolved this issue:
splunk migrate migrate-kvstore
this with the new certificate and I now dont have any issues.
Thank you for the reply
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Best way to fix the issue is:
1. Run the command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
2. Check the expiry date of output if expired then do the below steps:
3. Go to $SPLUNK_HOME\etc\auth\
4. Rename server.pem to server.pem_backup
5. Restart the splunk using command ./splunk restart
6. After restart you will be able to see a new server.pem file.
7. Check the expiry date of Certificate now using command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
8. The expiry date will be extended.
9. After restart the kvstore will be up and running.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks alot this works for me
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

This worked for me. after renaming the server.pem file, i restarted the service.
cd /opt/splunk/bin/
openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
output - notAfter=Feb 24 07:44:43 2025 GMT
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This worked for me - I'd mark this as correct answer! Thank you!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Worked for me too, thanks.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@tcmarquesi
Welcome 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@sphadnis
Can you please mark the answer. so the question can be closed.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
resolved this issue:
splunk migrate migrate-kvstore
this with the new certificate and I now dont have any issues.
Thank you for the reply
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

If you are running Search Head Clustering, **DO NOT ** follow the directions below. (Though they might guide you in the right direction.)
I recently had this same error in my lab environment. In my case, Splunk's internal SSL certificate simply expired. I thought it was related to an upgrade to Splunk 7.2.x, but it was just the passage of time.
Run this command to check if this is the case:
# openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
Example output showing it has expired:
notAfter=Oct 23 01:24:56 2018 GMT
To create a new cert, you can use your company's certificate server, or just use Splunk's createssl command:
$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048
Tailor the arguments as needed. Once done, re-run the command
# openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
Example output showing it has been renewed:
notAfter=Nov 12 18:37:53 2021 GMT
Then just restart Splunk and your Splunk KV Store should be working again.
Many thanks to jcrabb who wrote https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html
