Can you help me with the following mongod kvstore error?

[lhc_systems]

Hi All

I have recently taken over the admin of our splunk server, I upgraded to 7.2.0 and its been running fine for a while, yesterday we started getting errors:

Failed to start KV Store process. See mongod.log and splunkd.log for details.
11/13/2018, 9:09:18 AM
KV Store changed status to failed. KVStore process terminated.
11/13/2018, 9:09:16 AM
KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details.
11/13/2018, 9:09:16 AM

after looking that up I saw that the internal SSL cert had expired so I renewed it as per the instructions:

"set OPENSSL_CONF=D:\Splunk\openssl.cnf
D:\Splunk\etc\auth>d:\splunk\bin\splunk createssl server-cert -d . -n server"

This is now showing the cert to be valid. But now, I am getting the error below in the mongod log file.

2018-11-13T09:09:16.227Z W CONTROL  [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] MongoDB starting : pid=8104 port=8191 dbpath=E:\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=PRDSPLKAPP02
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] db version v3.6.7
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2o-fips  27 Mar 2018
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] allocator: tcmalloc
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] modules: none
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] build environment:
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     distmod: 2008plus-ssl
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     distarch: x86_64
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     target_arch: x86_64
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "E:\Splunk\etc\auth\server.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "C3E895A2-5F0A-4968-856E-C1C0047199B9" }, security: { javascriptEnabled: false, keyFile: "E:\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "E:\Splunk\var\lib\splunk\kvstore\mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
 2018-11-13T09:09:16.401Z I JOURNAL  [initandlisten] journal dir=E:\Splunk\var\lib\splunk\kvstore\mongo\journal
 2018-11-13T09:09:16.401Z I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
 2018-11-13T09:09:16.457Z I JOURNAL  [durability] Durability thread started
 2018-11-13T09:09:16.458Z I JOURNAL  [journal writer] Journal writer thread started
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] 
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] **          Please specify an sslCAFile parameter.
 2018-11-13T09:09:16.488Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
 2018-11-13T09:09:16.488Z I NETWORK  [initandlisten] shutdown: going to close listening sockets...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutdown: removing all drop-pending collections...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutting down replication subsystems
 2018-11-13T09:09:16.488Z W REPL     [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished.  Shutting down without cleaning up the replication system
 2018-11-13T09:09:16.488Z I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...
 2018-11-13T09:09:16.488Z I STORAGE  [initandlisten] shutdown: final commit...
 2018-11-13T09:09:16.492Z I JOURNAL  [initandlisten] journalCleanup...
 2018-11-13T09:09:16.492Z I JOURNAL  [initandlisten] removeJournalFiles
 2018-11-13T09:09:16.497Z I JOURNAL  [initandlisten] old journal file E:\Splunk\var\lib\splunk\kvstore\mongo\journal\j._0 will be reused as E:\Splunk\var\lib\splunk\kvstore\mongo\journal\prealloc.0
 2018-11-13T09:09:16.498Z I JOURNAL  [initandlisten] Terminating durability thread ...
 2018-11-13T09:09:16.521Z I JOURNAL  [journal writer] Journal writer thread stopped
 2018-11-13T09:09:16.521Z I JOURNAL  [durability] Durability thread stopped
 2018-11-13T09:09:16.521Z I STORAGE  [initandlisten] shutdown: closing all files...
 2018-11-13T09:09:16.534Z I STORAGE  [initandlisten] closeAllFiles() finished
 2018-11-13T09:09:16.534Z I STORAGE  [initandlisten] shutdown: removing fs lock...
 2018-11-13T09:09:16.535Z I CONTROL  [initandlisten] now exiting
 2018-11-13T09:09:16.535Z I CONTROL  [initandlisten] shutting down with code:62

The two big errors being:

"Please specify an sslCAFile parameter."

where do I specify this?

** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

would this not have upgraded with the version of Splunk? if not, how do I upgrade this?

any help would be appreciated, Thank you

[vishaltaneja070]

Best way to fix the issue is:
1. Run the command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
2. Check the expiry date of output if expired then do the below steps:
3. Go to $SPLUNK_HOME\etc\auth\
4. Rename server.pem to server.pem_backup
5. Restart the splunk using command ./splunk restart
6. After restart you will be able to see a new server.pem file.
7. Check the expiry date of Certificate now using command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
8. The expiry date will be extended.
9. After restart the kvstore will be up and running.

[ssuluguri]

Thanks alot this works for me

[aakif]

This worked for me. after renaming the server.pem file, i restarted the service.

cd /opt/splunk/bin/

openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

output - notAfter=Feb 24 07:44:43 2025 GMT

[sphadnis]

This worked for me - I'd mark this as correct answer! Thank you!

[tcmarquesi]

Worked for me too, thanks.

[vishaltaneja070]

@tcmarquesi
Welcome 🙂

[vishaltaneja070]

@sphadnis
Can you please mark the answer. so the question can be closed.

[lhc_systems · Accepted solution]

resolved this issue:

splunk migrate migrate-kvstore

this with the new certificate and I now dont have any issues.

Thank you for the reply

[mdonnelly_splun]

If you are running Search Head Clustering, **DO NOT ** follow the directions below. (Though they might guide you in the right direction.)

I recently had this same error in my lab environment. In my case, Splunk's internal SSL certificate simply expired. I thought it was related to an upgrade to Splunk 7.2.x, but it was just the passage of time.

Run this command to check if this is the case:

# openssl x509 -enddate -noout -in  $SPLUNK_HOME/etc/auth/server.pem

Example output showing it has expired:

notAfter=Oct 23 01:24:56 2018 GMT

To create a new cert, you can use your company's certificate server, or just use Splunk's createssl command:

$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048

Tailor the arguments as needed. Once done, re-run the command

# openssl x509 -enddate -noout -in  $SPLUNK_HOME/etc/auth/server.pem

Example output showing it has been renewed:

notAfter=Nov 12 18:37:53 2021 GMT

Then just restart Splunk and your Splunk KV Store should be working again.

Many thanks to jcrabb who wrote https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html