KV Store switching from failed to starting and stands there

chrkohm — Mon, 06 Jul 2020 06:08:23 GMT

Hi Guys,

I´m facing an issue with the KV Store.

The Status of the KV Store stands still on Starting:

This member: backupRestoreStatus : Ready disabled : 0 guid : 987EAE0D-F75D-401C-B21F-4E640CBC9019 port : 8191 standalone : 1 status : starting

The mongod.log says:

I ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

I already have set the permission on chmod 400 to the /mongo/splunk.key, but it still shows this message after restarting splunk

2020-07-06T05:34:24.156Z E - [rsSync] Assertion: 17322:write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200 src/mongo/db/repl/oplog.cpp 150 2020-07-06T05:34:24.156Z F - [rsSync] terminate() called. An exception is active; attempting to gather more information 2020-07-06T05:34:24.164Z F - [rsSync] DBException::toString(): Location17322: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200 Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException 0x55738d1dee21 0x55738d1de805 0x55738d2d36e6 0x55738d367b39 0x55738d2d3085 0x55738d370083 0x55738d3708d7 0x55738c05cdd1 0x55738bd33ee4 0x55738bd3468f 0x55738bcb0c4c 0x55738d2ee8a0 0x7fdfa06c4724 0x7fdfa0403e8d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55738AF1E000","o":"22C0E21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55738AF1E000","o":"22C0805"},{"b":"55738AF1E000","o":"23B56E6","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"55738AF1E000","o":"2449B39"},{"b":"55738AF1E000","o":"23B5085","s":"__gxx_personality_v0"},{"b":"55738AF1E000","o":"2452083"},{"b":"55738AF1E000","o":"24528D7"},{"b":"55738AF1E000","o":"113EDD1","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx"},{"b":"55738AF1E000","o":"E15EE4","s":"_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE"},{"b":"55738AF1E000","o":"E1668F","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE"},{"b":"55738AF1E000","o":"D92C4C","s":"_ZN5mongo4repl10RSDataSync4_runEv"},{"b":"55738AF1E000","o":"23D08A0"},{"b":"7FDFA06BC000","o":"8724"},{"b":"7FDFA0317000","o":"ECE8D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17-SERVER-42525-splunk", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.180-94.97-default", "version" : "#1 SMP Tue Jun 11 08:19:03 UTC 2019 (c0632f4)", "machine" : "x86_64" }, "somap" : [ { "b" : "55738AF1E000", "elfType" : 3 }, { "b" : "7FFF26F76000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7FDFA12C3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3 }, { "b" : "7FDFA0FE2000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA1669000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA0DDE000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FDFA0BD6000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FDFA08D9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FDFA06BC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FDFA0317000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FDFA14DA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FDFA164B000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55738d1dee21] mongod(+0x22C0805) [0x55738d1de805] mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55738d2d36e6] mongod(+0x2449B39) [0x55738d367b39] mongod(__gxx_personality_v0+0x2B5) [0x55738d2d3085] mongod(+0x2452083) [0x55738d370083] mongod(+0x24528D7) [0x55738d3708d7] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx+0x511) [0x55738c05cdd1] mongod(_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE+0xC04) [0x55738bd33ee4] mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE+0x13F) [0x55738bd3468f] mongod(_ZN5mongo4repl10RSDataSync4_runEv+0x11C) [0x55738bcb0c4c] mongod(+0x23D08A0) [0x55738d2ee8a0] libpthread.so.0(+0x8724) [0x7fdfa06c4724] libc.so.6(clone+0x6D) [0x7fdfa0403e8d] ----- END BACKTRACE -----

Could please someone help me with this?

Re: KV Store switching from failed to starting and stands there

chrkohm — Mon, 06 Jul 2020 06:49:10 GMT

./splunk stop

./splunk clean kvstore --local

./splunk start

Solves the issue for me.

topic Re: KV Store switching from failed to starting and stands there in Installation

KV Store switching from failed to starting and stands there

Re: KV Store switching from failed to starting and stands there