Installation

KV Store switching from failed to starting and stands there

chrkohm
Path Finder

Hi Guys,

I´m facing an issue with the KV Store.

 

The Status of the KV Store stands still on Starting:

 

This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 987EAE0D-F75D-401C-B21F-4E640CBC9019
                                          port : 8191
                                    standalone : 1
                                        status : starting

 

The mongod.log says:

 

I ACCESS   [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

I already have set the permission on chmod 400 to the /mongo/splunk.key, but it still shows this message after restarting splunk

 

 2020-07-06T05:34:24.156Z E -        [rsSync] Assertion: 17322:write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200 src/mongo/db/repl/oplog.cpp 150
 2020-07-06T05:34:24.156Z F -        [rsSync] terminate() called. An exception is active; attempting to gather more information
 2020-07-06T05:34:24.164Z F -        [rsSync] DBException::toString(): Location17322: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:209715200
 Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException
  0x55738d1dee21 0x55738d1de805 0x55738d2d36e6 0x55738d367b39 0x55738d2d3085 0x55738d370083 0x55738d3708d7 0x55738c05cdd1 0x55738bd33ee4 0x55738bd3468f 0x55738bcb0c4c 0x55738d2ee8a0 0x7fdfa06c4724 0x7fdfa0403e8d
 ----- BEGIN BACKTRACE -----
 {"backtrace":[{"b":"55738AF1E000","o":"22C0E21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55738AF1E000","o":"22C0805"},{"b":"55738AF1E000","o":"23B56E6","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"55738AF1E000","o":"2449B39"},{"b":"55738AF1E000","o":"23B5085","s":"__gxx_personality_v0"},{"b":"55738AF1E000","o":"2452083"},{"b":"55738AF1E000","o":"24528D7"},{"b":"55738AF1E000","o":"113EDD1","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx"},{"b":"55738AF1E000","o":"E15EE4","s":"_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE"},{"b":"55738AF1E000","o":"E1668F","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE"},{"b":"55738AF1E000","o":"D92C4C","s":"_ZN5mongo4repl10RSDataSync4_runEv"},{"b":"55738AF1E000","o":"23D08A0"},{"b":"7FDFA06BC000","o":"8724"},{"b":"7FDFA0317000","o":"ECE8D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17-SERVER-42525-splunk", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.180-94.97-default", "version" : "#1 SMP Tue Jun 11 08:19:03 UTC 2019 (c0632f4)", "machine" : "x86_64" }, "somap" : [ { "b" : "55738AF1E000", "elfType" : 3 }, { "b" : "7FFF26F76000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7FDFA12C3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3 }, { "b" : "7FDFA0FE2000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA1669000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FDFA0DDE000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FDFA0BD6000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FDFA08D9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FDFA06BC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FDFA0317000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FDFA14DA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FDFA164B000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55738d1dee21]
  mongod(+0x22C0805) [0x55738d1de805]
  mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55738d2d36e6]
  mongod(+0x2449B39) [0x55738d367b39]
  mongod(__gxx_personality_v0+0x2B5) [0x55738d2d3085]
  mongod(+0x2452083) [0x55738d370083]
  mongod(+0x24528D7) [0x55738d3708d7]
  mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx+0x511) [0x55738c05cdd1]
  mongod(_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE+0xC04) [0x55738bd33ee4]
  mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_22ReplicationCoordinatorE+0x13F) [0x55738bd3468f]
  mongod(_ZN5mongo4repl10RSDataSync4_runEv+0x11C) [0x55738bcb0c4c]
  mongod(+0x23D08A0) [0x55738d2ee8a0]
  libpthread.so.0(+0x8724) [0x7fdfa06c4724]
  libc.so.6(clone+0x6D) [0x7fdfa0403e8d]
 -----  END BACKTRACE  -----

 

Could please someone help me with this?

Labels (2)
Tags (2)
0 Karma
1 Solution

chrkohm
Path Finder

./splunk stop

./splunk clean kvstore --local

./splunk start

 

Solves the issue for me.

View solution in original post

Tags (2)
0 Karma

chrkohm
Path Finder

./splunk stop

./splunk clean kvstore --local

./splunk start

 

Solves the issue for me.

Tags (2)
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...