Splunk Enterprise

kvstore failed after downgrade from 8.0.2 to 7.0.2

leoneed
Engager

Hi all!

While testing rollback workflow we faced with kvstore failed. When we try to start splunk with ./splunk start we get followed error in var/log/splunk/mongod.log:

 

2020-06-30T16:42:40.231Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] MongoDB starting : pid=120021 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=vm08
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] db version v3.0.14-splunk
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] git version: 08352afcca24bfc145240a0fac9d28b978ab77f3
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] build info: Linux ip-10-113-204-203 2.6.18-194.el5xen #1 SMP Tue Mar 16 22:01:26 EDT 2010 x86_64 BOOST_LIB_VERSION=1_49
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] allocator: tcmalloc
2020-06-30T16:42:40.296Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "0F92C83E-3832-4718-8A20-6AE05900C13D" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2020-06-30T16:42:40.350Z W - [initandlisten] Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty.
2020-06-30T16:42:40.391Z I STORAGE [initandlisten]
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2020-06-30T16:42:40.391Z I STORAGE [initandlisten] ** http://dochub.mongodb.org/core/readahead
2020-06-30T16:42:40.392Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2020-06-30T16:42:40.392Z I JOURNAL [initandlisten] recover begin
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] info no lsn file in journal/ directory
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] recover lsn: 0
2020-06-30T16:42:40.393Z I JOURNAL [initandlisten] recover /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0
2020-06-30T16:42:40.397Z I JOURNAL [initandlisten] recover cleaning up
2020-06-30T16:42:40.397Z I JOURNAL [initandlisten] removeJournalFiles
2020-06-30T16:42:40.439Z I JOURNAL [initandlisten] recover done
2020-06-30T16:42:40.439Z I JOURNAL [initandlisten] preallocating a journal file /opt/splunk/var/lib/splunk/kvstore/mongo/journal/prealloc.0
2020-06-30T16:42:41.261Z I JOURNAL [durability] Durability thread started
2020-06-30T16:42:41.278Z I JOURNAL [journal writer] Journal writer thread started
2020-06-30T16:42:41.371Z I - [initandlisten] Invariant failure 1 == version src/mongo/db/storage/mmap_v1/btree/btree_interface.cpp 267
2020-06-30T16:42:41.391Z I CONTROL [initandlisten]
0x55fcf57642e2 0x55fcf56fea69 0x55fcf56e1b39 0x55fcf54e36be 0x55fcf552b674 0x55fcf50db490 0x55fcf50deb20 0x55fcf50c8cc1 0x55fcf50d3f53 0x55fcf50d5f36 0x55fcf50d8db6 0x55fcf4fb410d 0x55fcf4fb8a09 0x7fe558f44555 0x55fcf4fb1149
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55FCF4A99000","o":"CCB2E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FCF4A99000","o":"C65A69","s":"_ZN5mongo10logContextEPKc"},{"b":"55FCF4A99000","o":"C48B39","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"55FCF4A99000","o":"A4A6BE","s":"_ZN5mongo18getMMAPV1InterfaceEPNS_11HeadManagerEPNS_11RecordStoreEPNS_19SavedCursorRegistryERKNS_8OrderingERKSsi"},{"b":"55FCF4A99000","o":"A92674","s":"_ZN5mongo26MMAPV1DatabaseCatalogEntry8getIndexEPNS_16OperationContextEPKNS_22CollectionCatalogEntryEPNS_17IndexCatalogEntryE"},{"b":"55FCF4A99000","o":"642490","s":"_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_16OperationContextEPNS_15IndexDescriptorEb"},{"b":"55FCF4A99000","o":"645B20","s":"_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE"},{"b":"55FCF4A99000","o":"62FCC1","s":"_ZN5mongo10CollectionC2EPNS_16OperationContextERKNS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE"},{"b":"55FCF4A99000","o":"63AF53","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE"},{"b":"55FCF4A99000","o":"63CF36","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"55FCF4A99000","o":"63FDB6","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb"},{"b":"55FCF4A99000","o":"51B10D","s":"_ZN5mongo13initAndListenEi"},{"b":"55FCF4A99000","o":"51FA09","s":"main"},{"b":"7FE558F22000","o":"22555","s":"__libc_start_main"},{"b":"55FCF4A99000","o":"518149"}],"processInfo":{ "mongodbVersion" : "3.0.14-splunk", "gitVersion" : "08352afcca24bfc145240a0fac9d28b978ab77f3", "uname" : { "sysname" : "Linux", "release" : "3.10.0-1062.18.1.el7.x86_64", "version" : "#1 SMP Tue Mar 17 23:49:17 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "55FCF4A99000", "elfType" : 3 }, { "b" : "7FFD806E9000", "elfType" : 3 }, { "b" : "7FE559CD9000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FE55A099000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FE5599FE000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FE5597F6000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FE5595F2000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FE5592F0000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FE558F22000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FE559EF5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FE55A07B000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x55fcf57642e2]
mongod(_ZN5mongo10logContextEPKc+0xE9) [0x55fcf56fea69]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB9) [0x55fcf56e1b39]
mongod(_ZN5mongo18getMMAPV1InterfaceEPNS_11HeadManagerEPNS_11RecordStoreEPNS_19SavedCursorRegistryERKNS_8OrderingERKSsi+0x1CE) [0x55fcf54e36be]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry8getIndexEPNS_16OperationContextEPKNS_22CollectionCatalogEntryEPNS_17IndexCatalogEntryE+0x74) [0x55fcf552b674]
mongod(_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_16OperationContextEPNS_15IndexDescriptorEb+0x90) [0x55fcf50db490]
mongod(_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE+0x260) [0x55fcf50deb20]
mongod(_ZN5mongo10CollectionC2EPNS_16OperationContextERKNS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE+0x111) [0x55fcf50c8cc1]
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0x93) [0x55fcf50d3f53]
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x206) [0x55fcf50d5f36]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x176) [0x55fcf50d8db6]
mongod(_ZN5mongo13initAndListenEi+0x107D) [0x55fcf4fb410d]
mongod(main+0x159) [0x55fcf4fb8a09]
libc.so.6(__libc_start_main+0xF5) [0x7fe558f44555]
mongod(+0x518149) [0x55fcf4fb1149]
----- END BACKTRACE -----
2020-06-30T16:42:41.391Z I - [initandlisten]
***aborting after invariant() failure

 

please advise.

how to make a downgrade correctly?

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

I would suggest you read (or re-read) https://docs.splunk.com/Documentation/Splunk/8.0.4/Installation/AboutupgradingREADTHISFIRST

Use Gemini KV Store Tools or in newer versions of Splunk you can do the kvstore backup directly

I do not believe rollback is officially supported but the way I do it is to delete directories which I know are changed during an upgrade.

In this case you might need to wipe your kvstore directories and restore from backup

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Maybe this helps you https://community.splunk.com/t5/Knowledge-Management/After-upgrading-to-6-5-0-KV-Store-will-not-star...
It seems that your ssl config has corrupted/missed.

R. Ismo
0 Karma

gjanders
SplunkTrust
SplunkTrust

I would suggest you read (or re-read) https://docs.splunk.com/Documentation/Splunk/8.0.4/Installation/AboutupgradingREADTHISFIRST

Use Gemini KV Store Tools or in newer versions of Splunk you can do the kvstore backup directly

I do not believe rollback is officially supported but the way I do it is to delete directories which I know are changed during an upgrade.

In this case you might need to wipe your kvstore directories and restore from backup

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...