Knowledge Management

Can you help me with the following mongod kvstore error?

lhc_systems
Engager

Hi All

I have recently taken over the admin of our splunk server, I upgraded to 7.2.0 and its been running fine for a while, yesterday we started getting errors:

Failed to start KV Store process. See mongod.log and splunkd.log for details.
11/13/2018, 9:09:18 AM
KV Store changed status to failed. KVStore process terminated.
11/13/2018, 9:09:16 AM
KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details.
11/13/2018, 9:09:16 AM

after looking that up I saw that the internal SSL cert had expired so I renewed it as per the instructions:

"set OPENSSL_CONF=D:\Splunk\openssl.cnf
D:\Splunk\etc\auth>d:\splunk\bin\splunk createssl server-cert -d . -n server"

This is now showing the cert to be valid. But now, I am getting the error below in the mongod log file.

2018-11-13T09:09:16.227Z W CONTROL  [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] MongoDB starting : pid=8104 port=8191 dbpath=E:\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=PRDSPLKAPP02
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] db version v3.6.7
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2o-fips  27 Mar 2018
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] allocator: tcmalloc
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] modules: none
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] build environment:
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     distmod: 2008plus-ssl
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     distarch: x86_64
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten]     target_arch: x86_64
 2018-11-13T09:09:16.379Z I CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "E:\Splunk\etc\auth\server.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "C3E895A2-5F0A-4968-856E-C1C0047199B9" }, security: { javascriptEnabled: false, keyFile: "E:\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "E:\Splunk\var\lib\splunk\kvstore\mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
 2018-11-13T09:09:16.401Z I JOURNAL  [initandlisten] journal dir=E:\Splunk\var\lib\splunk\kvstore\mongo\journal
 2018-11-13T09:09:16.401Z I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
 2018-11-13T09:09:16.457Z I JOURNAL  [durability] Durability thread started
 2018-11-13T09:09:16.458Z I JOURNAL  [journal writer] Journal writer thread started
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] 
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
 2018-11-13T09:09:16.460Z I CONTROL  [initandlisten] **          Please specify an sslCAFile parameter.
 2018-11-13T09:09:16.488Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
 2018-11-13T09:09:16.488Z I NETWORK  [initandlisten] shutdown: going to close listening sockets...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutdown: removing all drop-pending collections...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutdown: removing checkpointTimestamp collection...
 2018-11-13T09:09:16.488Z I REPL     [initandlisten] shutting down replication subsystems
 2018-11-13T09:09:16.488Z W REPL     [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished.  Shutting down without cleaning up the replication system
 2018-11-13T09:09:16.488Z I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...
 2018-11-13T09:09:16.488Z I STORAGE  [initandlisten] shutdown: final commit...
 2018-11-13T09:09:16.492Z I JOURNAL  [initandlisten] journalCleanup...
 2018-11-13T09:09:16.492Z I JOURNAL  [initandlisten] removeJournalFiles
 2018-11-13T09:09:16.497Z I JOURNAL  [initandlisten] old journal file E:\Splunk\var\lib\splunk\kvstore\mongo\journal\j._0 will be reused as E:\Splunk\var\lib\splunk\kvstore\mongo\journal\prealloc.0
 2018-11-13T09:09:16.498Z I JOURNAL  [initandlisten] Terminating durability thread ...
 2018-11-13T09:09:16.521Z I JOURNAL  [journal writer] Journal writer thread stopped
 2018-11-13T09:09:16.521Z I JOURNAL  [durability] Durability thread stopped
 2018-11-13T09:09:16.521Z I STORAGE  [initandlisten] shutdown: closing all files...
 2018-11-13T09:09:16.534Z I STORAGE  [initandlisten] closeAllFiles() finished
 2018-11-13T09:09:16.534Z I STORAGE  [initandlisten] shutdown: removing fs lock...
 2018-11-13T09:09:16.535Z I CONTROL  [initandlisten] now exiting
 2018-11-13T09:09:16.535Z I CONTROL  [initandlisten] shutting down with code:62

The two big errors being:

"Please specify an sslCAFile parameter."

where do I specify this?

** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

would this not have upgraded with the version of Splunk? if not, how do I upgrade this?

any help would be appreciated, Thank you

0 Karma
1 Solution

lhc_systems
Engager

resolved this issue:

splunk migrate migrate-kvstore

this with the new certificate and I now dont have any issues.

Thank you for the reply

View solution in original post

0 Karma

vishaltaneja070
Motivator

Best way to fix the issue is:
1. Run the command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
2. Check the expiry date of output if expired then do the below steps:
3. Go to $SPLUNK_HOME\etc\auth\
4. Rename server.pem to server.pem_backup
5. Restart the splunk using command ./splunk restart
6. After restart you will be able to see a new server.pem file.
7. Check the expiry date of Certificate now using command: $SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem
8. The expiry date will be extended.
9. After restart the kvstore will be up and running.

ssuluguri
Path Finder

Thanks alot this works for me

0 Karma

aakif
Engager

This worked for me. after renaming the server.pem file, i restarted the service.

 

cd /opt/splunk/bin/

openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

output - notAfter=Feb 24 07:44:43 2025 GMT

0 Karma

sphadnis
Path Finder

This worked for me - I'd mark this as correct answer! Thank you!

0 Karma

tcmarquesi
Explorer

Worked for me too, thanks.

0 Karma

vishaltaneja070
Motivator

@tcmarquesi
Welcome 🙂

0 Karma

vishaltaneja070
Motivator

@sphadnis
Can you please mark the answer. so the question can be closed.

0 Karma

lhc_systems
Engager

resolved this issue:

splunk migrate migrate-kvstore

this with the new certificate and I now dont have any issues.

Thank you for the reply

0 Karma

mdonnelly_splun
Splunk Employee
Splunk Employee

If you are running Search Head Clustering, **DO NOT ** follow the directions below. (Though they might guide you in the right direction.)

I recently had this same error in my lab environment. In my case, Splunk's internal SSL certificate simply expired. I thought it was related to an upgrade to Splunk 7.2.x, but it was just the passage of time.

Run this command to check if this is the case:

# openssl x509 -enddate -noout -in  $SPLUNK_HOME/etc/auth/server.pem

Example output showing it has expired:

notAfter=Oct 23 01:24:56 2018 GMT

To create a new cert, you can use your company's certificate server, or just use Splunk's createssl command:

$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048

Tailor the arguments as needed. Once done, re-run the command

# openssl x509 -enddate -noout -in  $SPLUNK_HOME/etc/auth/server.pem

Example output showing it has been renewed:

notAfter=Nov 12 18:37:53 2021 GMT

Then just restart Splunk and your Splunk KV Store should be working again.

Many thanks to jcrabb who wrote https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...