Installation

Why is the KV Store Process Failing After Upgrade from 7.1.3 to 7.2.0?

hobbymaster001
Engager

I upgraded from 7.1.3 to 7.2.0 as stated in the title, there were no errors with the upgrade and everything seemed to start alright. Upon logging in I was greeted with the following three messages:

  • Failed to start KV Store process. See mongod.log and splunkd.log for details. 10/2/2018, 4:38:28 PM
  • KV Store changed status to failed. KVStore process terminated. 10/2/2018, 4:38:27 PM
  • KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details. 10/2/2018, 4:38:27 PM

Looking through the mongod.log file I found the following:

 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten]
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 2018-10-02T20:38:27.220Z I JOURNAL  [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
 2018-10-02T20:38:27.220Z I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
 2018-10-02T20:38:27.267Z I JOURNAL  [durability] Durability thread started
 2018-10-02T20:38:27.267Z I JOURNAL  [journal writer] Journal writer thread started
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten] **          Please specify an sslCAFile parameter.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] **        We suggest setting it to 'never'
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] **        We suggest setting it to 'never'
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.319Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

After the IMPORTANT tag the DB process exits. Does anyone know how to either manually upgrade the database to 3.6 or how to purge and recreate it? I cannot seem to locate any information on this. The troubleshooting page wasn't a help in this instance as I already knew the service was failed and likely why. I have verified that the path to the DB is correct as well and verified permissions were alright. I also tried deleting the lock file and restarting to find the same error message.

Thanks for any help!

Labels (1)
1 Solution

dauren_akilbeko
Communicator

santosh_sshanbh
Path Finder

This command also solved the same issue for me. But what exactly this command does?

0 Karma

rkirkw
Path Finder

Thx for this post - got my KV store going again. It seems like the upgrade to 7.2.x would have done this step for you since it was a required step!

0 Karma

michaelgilbert
Engager

Had this same issue and this solution solved it for me! Thanks for the post!

0 Karma

Hoekb03
Explorer

Same issue here with a previous update. This seems to have fixed it. Thx!

0 Karma

hobbymaster001
Engager

Thank you so much for that! I didn't see that on the troubleshooting page when I looked through it and my issue was exactly listed... I guess I need to read closer. It took about 2 seconds to fix and is reporting a-ok now.

0 Karma

0YAoNnmRmKDg
Path Finder

Awesome, thank you too! - fixed

    10/4/18
7:44:27.882 PM  
 2018-10-04T06:44:27.882Z I CONTROL  [initandlisten] MongoDB starting : pid=4329 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:15:55.506 PM  
 2018-10-04T05:15:55.506Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:15:22.876 PM  
 2018-10-04T05:15:22.876Z I CONTROL  [initandlisten] MongoDB starting : pid=18818 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:08:03.372 PM  
 2018-10-04T05:08:03.372Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:08:01.914 PM  
 2018-10-04T05:08:01.914Z I CONTROL  [initandlisten] MongoDB starting : pid=11974 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  
0 Karma

dauren_akilbeko
Communicator

Yeah had the same problem yesterday on my test server and found the solution in docs, so I knew where to look 🙂

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...