kvstore failed after downgrade from 8.0.2 to 7.0.2

[leoneed]

Hi all!

While testing rollback workflow we faced with kvstore failed. When we try to start splunk with ./splunk start we get followed error in var/log/splunk/mongod.log:

2020-06-30T16:42:40.231Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] MongoDB starting : pid=120021 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=vm08
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] db version v3.0.14-splunk
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] git version: 08352afcca24bfc145240a0fac9d28b978ab77f3
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] build info: Linux ip-10-113-204-203 2.6.18-194.el5xen #1 SMP Tue Mar 16 22:01:26 EDT 2010 x86_64 BOOST_LIB_VERSION=1_49
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] allocator: tcmalloc
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "0F92C83E-3832-4718-8A20-6AE05900C13D" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2020-06-30T16:42:40.350Z W - [initandlisten] Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty.
2020-06-30T16:42:40.391Z I STORAGE [initandlisten]
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** http://dochub.mongodb.org/core/readahead
2020-06-30T16:42:40.392Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2020-06-30T16:42:40.392Z I JOURNAL [initandlisten] recover begin
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] info no lsn file in journal/ directory
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] recover lsn: 0
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] recover /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0
2020-06-30T16:42:40.397Z I JOURNAL [initandlisten] recover cleaning up
2020-06-30T16:42:40.397Z I JOURNAL [initandlisten] removeJournalFiles
2020-06-30T16:42:40.439Z I JOURNAL [initandlisten] recover done
2020-06-30T16:42:40.439Z I JOURNAL [initandlisten] preallocating a journal file /opt/splunk/var/lib/splunk/kvstore/mongo/journal/prealloc.0
2020-06-30T16:42:41.261Z I JOURNAL [durability] Durability thread started
2020-06-30T16:42:41.278Z I JOURNAL [journal writer] Journal writer thread started
2020-06-30T16:42:41.371Z I - [initandlisten] Invariant failure 1 == version src/mongo/db/storage/mmap_v1/btree/btree_interface.cpp 267
2020-06-30T16:42:41.391Z I CONTROL [initandlisten]
0x55fcf57642e2 0x55fcf56fea69 0x55fcf56e1b39 0x55fcf54e36be 0x55fcf552b674 0x55fcf50db490 0x55fcf50deb20 0x55fcf50c8cc1 0x55fcf50d3f53 0x55fcf50d5f36 0x55fcf50d8db6 0x55fcf4fb410d 0x55fcf4fb8a09 0x7fe558f44555 0x55fcf4fb1149
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55FCF4A99000","o":"CCB2E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FCF4A99000","o":"C65A69","s":"_ZN5mongo10logContextEPKc"},{"b":"55FCF4A99000","o":"C48B39","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"55FCF4A99000","o":"A4A6BE","s":"_ZN5mongo18getMMAPV1InterfaceEPNS_11HeadManagerEPNS_11RecordStoreEPNS_19SavedCursorRegistryERKNS_8OrderingERKSsi"},{"b":"55FCF4A99000","o":"A92674","s":"_ZN5mongo26MMAPV1DatabaseCatalogEntry8getIndexEPNS_16OperationContextEPKNS_22CollectionCatalogEntryEPNS_17IndexCatalogEntryE"},{"b":"55FCF4A99000","o":"642490","s":"_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_16OperationContextEPNS_15IndexDescriptorEb"},{"b":"55FCF4A99000","o":"645B20","s":"_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE"},{"b":"55FCF4A99000","o":"62FCC1","s":"_ZN5mongo10CollectionC2EPNS_16OperationContextERKNS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE"},{"b":"55FCF4A99000","o":"63AF53","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE"},{"b":"55FCF4A99000","o":"63CF36","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"55FCF4A99000","o":"63FDB6","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb"},{"b":"55FCF4A99000","o":"51B10D","s":"_ZN5mongo13initAndListenEi"},{"b":"55FCF4A99000","o":"51FA09","s":"main"},{"b":"7FE558F22000","o":"22555","s":"__libc_start_main"},{"b":"55FCF4A99000","o":"518149"}],"processInfo":{ "mongodbVersion" : "3.0.14-splunk", "gitVersion" : "08352afcca24bfc145240a0fac9d28b978ab77f3", "uname" : { "sysname" : "Linux", "release" : "3.10.0-1062.18.1.el7.x86_64", "version" : "#1 SMP Tue Mar 17 23:49:17 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "55FCF4A99000", "elfType" : 3 }, { "b" : "7FFD806E9000", "elfType" : 3 }, { "b" : "7FE559CD9000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FE55A099000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FE5599FE000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FE5597F6000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FE5595F2000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FE5592F0000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FE558F22000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FE559EF5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FE55A07B000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x55fcf57642e2]
mongod(_ZN5mongo10logContextEPKc+0xE9) [0x55fcf56fea69]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB9) [0x55fcf56e1b39]
mongod(_ZN5mongo18getMMAPV1InterfaceEPNS_11HeadManagerEPNS_11RecordStoreEPNS_19SavedCursorRegistryERKNS_8OrderingERKSsi+0x1CE) [0x55fcf54e36be]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry8getIndexEPNS_16OperationContextEPKNS_22CollectionCatalogEntryEPNS_17IndexCatalogEntryE+0x74) [0x55fcf552b674]
mongod(_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_16OperationContextEPNS_15IndexDescriptorEb+0x90) [0x55fcf50db490]
mongod(_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE+0x260) [0x55fcf50deb20]
mongod(_ZN5mongo10CollectionC2EPNS_16OperationContextERKNS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE+0x111) [0x55fcf50c8cc1]
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0x93) [0x55fcf50d3f53]
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x206) [0x55fcf50d5f36]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x176) [0x55fcf50d8db6]
mongod(_ZN5mongo13initAndListenEi+0x107D) [0x55fcf4fb410d]
mongod(main+0x159) [0x55fcf4fb8a09]
libc.so.6(__libc_start_main+0xF5) [0x7fe558f44555]
mongod(+0x518149) [0x55fcf4fb1149]
----- END BACKTRACE -----
2020-06-30T16:42:41.391Z I - [initandlisten]
***aborting after invariant() failure

please advise.

how to make a downgrade correctly?

[isoutamo]

Maybe this helps you https://community.splunk.com/t5/Knowledge-Management/After-upgrading-to-6-5-0-KV-Store-will-not-star...
It seems that your ssl config has corrupted/missed.

R. Ismo

[gjanders · Accepted solution]

I would suggest you read (or re-read) https://docs.splunk.com/Documentation/Splunk/8.0.4/Installation/AboutupgradingREADTHISFIRST

Use Gemini KV Store Tools or in newer versions of Splunk you can do the kvstore backup directly

I do not believe rollback is officially supported but the way I do it is to delete directories which I know are changed during an upgrade.

In this case you might need to wipe your kvstore directories and restore from backup