Why is the KV Store Process Failing After Upgrade from 7.1.3 to 7.2.0?

[hobbymaster001]

I upgraded from 7.1.3 to 7.2.0 as stated in the title, there were no errors with the upgrade and everything seemed to start alright. Upon logging in I was greeted with the following three messages:

• Failed to start KV Store process. See mongod.log and splunkd.log for details. 10/2/2018, 4:38:28 PM
• KV Store changed status to failed. KVStore process terminated. 10/2/2018, 4:38:27 PM
• KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details. 10/2/2018, 4:38:27 PM

Looking through the mongod.log file I found the following:

 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten]
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 2018-10-02T20:38:27.219Z I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 2018-10-02T20:38:27.220Z I JOURNAL  [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
 2018-10-02T20:38:27.220Z I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
 2018-10-02T20:38:27.267Z I JOURNAL  [durability] Durability thread started
 2018-10-02T20:38:27.267Z I JOURNAL  [journal writer] Journal writer thread started
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
 2018-10-02T20:38:27.292Z I CONTROL  [initandlisten] **          Please specify an sslCAFile parameter.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] **        We suggest setting it to 'never'
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten] **        We suggest setting it to 'never'
 2018-10-02T20:38:27.293Z I CONTROL  [initandlisten]
 2018-10-02T20:38:27.319Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

After the IMPORTANT tag the DB process exits. Does anyone know how to either manually upgrade the database to 3.6 or how to purge and recreate it? I cannot seem to locate any information on this. The troubleshooting page wasn't a help in this instance as I already knew the service was failed and likely why. I have verified that the path to the DB is correct as well and verified permissions were alright. I also tried deleting the lock file and restarting to find the same error message.

Thanks for any help!

[dauren_akilbeko · Accepted solution]

Use splunk migrate migrate-kvstore

http://docs.splunk.com/Documentation/Splunk/7.2.0/Admin/TroubleshootKVstore

[santosh_sshanbh]

This command also solved the same issue for me. But what exactly this command does?

[rkirkw]

Thx for this post - got my KV store going again. It seems like the upgrade to 7.2.x would have done this step for you since it was a required step!

[michaelgilbert]

Had this same issue and this solution solved it for me! Thanks for the post!

[Hoekb03]

Same issue here with a previous update. This seems to have fixed it. Thx!

[hobbymaster001]

Thank you so much for that! I didn't see that on the troubleshooting page when I looked through it and my issue was exactly listed... I guess I need to read closer. It took about 2 seconds to fix and is reporting a-ok now.

[0YAoNnmRmKDg]

Awesome, thank you too! - fixed

    10/4/18
7:44:27.882 PM  
 2018-10-04T06:44:27.882Z I CONTROL  [initandlisten] MongoDB starting : pid=4329 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:15:55.506 PM  
 2018-10-04T05:15:55.506Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:15:22.876 PM  
 2018-10-04T05:15:22.876Z I CONTROL  [initandlisten] MongoDB starting : pid=18818 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:08:03.372 PM  
 2018-10-04T05:08:03.372Z F CONTROL  [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

    10/4/18
6:08:01.914 PM  
 2018-10-04T05:08:01.914Z I CONTROL  [initandlisten] MongoDB starting : pid=11974 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=splunk

    host =  splunk  
    index = _internal   
    source =    /opt/splunk/var/log/splunk/mongod.log   
    sourcetype =    mongod  

[dauren_akilbeko]

Yeah had the same problem yesterday on my test server and found the solution in docs, so I knew where to look 🙂