KV Store switching from failed to starting and stands there

[chrkohm]

Hi Guys,

I´m facing an issue with the KV Store.

The Status of the KV Store stands still on Starting:

This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 987EAE0D-F75D-401C-B21F-4E640CBC9019
                                          port : 8191
                                    standalone : 1
                                        status : starting

The mongod.log says:

I ACCESS   [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

I already have set the permission on chmod 400 to the /mongo/splunk.key, but it still shows this message after restarting splunk

 2020-07-06T05:34:24.156Z E -        [rsSync] Assertion: 17322:write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200 src/mongo/db/repl/oplog.cpp 150
 2020-07-06T05:34:24.156Z F -        [rsSync] terminate() called. An exception is active; attempting to gather more information
 2020-07-06T05:34:24.164Z F -        [rsSync] DBException::toString(): Location17322: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200
 Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException
  0x55738d1dee21 0x55738d1de805 0x55738d2d36e6 0x55738d367b39 0x55738d2d3085 0x55738d370083 0x55738d3708d7 0x55738c05cdd1 0x55738bd33ee4 0x55738bd3468f 0x55738bcb0c4c 0x55738d2ee8a0 0x7fdfa06c4724 0x7fdfa0403e8d
 ----- BEGIN BACKTRACE -----
 {"backtrace":[{"b":"55738AF1E000","o":"22C0E21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55738AF1E000","o":"22C0805"},{"b":"55738AF1E000","o":"23B56E6","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"55738AF1E000","o":"2449B39"},{"b":"55738AF1E000","o":"23B5085","s":"__gxx_personality_v0"},{"b":"55738AF1E000","o":"2452083"},{"b":"55738AF1E000","o":"24528D7"},{"b":"55738AF1E000","o":"113EDD1","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx"},{"b":"55738AF1E000","o":"E15EE4","s":"_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE"},{"b":"55738AF1E000","o":"E1668F","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE"},{"b":"55738AF1E000","o":"D92C4C","s":"_ZN5mongo4repl10RSDataSync4_runEv"},{"b":"55738AF1E000","o":"23D08A0"},{"b":"7FDFA06BC000","o":"8724"},{"b":"7FDFA0317000","o":"ECE8D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17-SERVER-42525-splunk", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.180-94.97-default", "version" : "#1 SMP Tue Jun 11 08:19:03 UTC 2019 (c0632f4)", "machine" : "x86_64" }, "somap" : [ { "b" : "55738AF1E000", "elfType" : 3 }, { "b" : "7FFF26F76000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7FDFA12C3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3 }, { "b" : "7FDFA0FE2000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA1669000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA0DDE000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FDFA0BD6000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FDFA08D9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FDFA06BC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FDFA0317000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FDFA14DA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FDFA164B000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55738d1dee21]
  mongod(+0x22C0805) [0x55738d1de805]
  mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55738d2d36e6]
  mongod(+0x2449B39) [0x55738d367b39]
  mongod(__gxx_personality_v0+0x2B5) [0x55738d2d3085]
  mongod(+0x2452083) [0x55738d370083]
  mongod(+0x24528D7) [0x55738d3708d7]
  mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx+0x511) [0x55738c05cdd1]
  mongod(_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE+0xC04) [0x55738bd33ee4]
  mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE+0x13F) [0x55738bd3468f]
  mongod(_ZN5mongo4repl10RSDataSync4_runEv+0x11C) [0x55738bcb0c4c]
  mongod(+0x23D08A0) [0x55738d2ee8a0]
  libpthread.so.0(+0x8724) [0x7fdfa06c4724]
  libc.so.6(clone+0x6D) [0x7fdfa0403e8d]
 -----  END BACKTRACE  -----

Could please someone help me with this?

[chrkohm · Accepted solution]

./splunk stop

./splunk clean kvstore --local

./splunk start

Solves the issue for me.