kvstore fails to start

[merrelr]

After from 7.2.1 to 8.6.1 I'm getting the following errors in mongo.log.

2021-10-16T15:01:30.798Z W CONTROL  [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] MongoDB starting : pid=2763 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=7d49a1b4a62a
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] db version v3.6.17-linux-splunk-v4
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] git version: 226949cc252af265483afbf859b446590b09b098
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2y-fips  16 Feb 2021
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] allocator: tcmalloc
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] modules: none
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] build environment:
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten]     distarch: x86_64
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten]     target_arch: x86_64
 2021-10-16T15:01:30.840Z I CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "BD6F6380-BE84-4A28-A93F-359500FA793C" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
 2021-10-16T15:01:30.858Z I JOURNAL  [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
 2021-10-16T15:01:30.860Z I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
 2021-10-16T15:01:30.863Z I CONTROL  [initandlisten] LogFile::synchronousAppend failed with 8192 bytes unwritten out of 8192 bytes;  b=0x562db354c000 Bad address
 2021-10-16T15:01:30.863Z F -        [initandlisten] Fatal Assertion 13515 at src/mongo/db/storage/mmap_v1/logfile.cpp 250
 2021-10-16T15:01:30.863Z F -        [initandlisten]
 ***aborting after fassert() failure
 2021-10-16T15:01:30.880Z F -        [initandlisten] Got signal: 6 (Aborted).
  0x562db04d2de1 0x562db04d1ff9 0x562db04d24dd 0x1466a4e77340 0x1466a4ad7cc9 0x1466a4adb0d8 0x562daeb8c792 0x562daf0f1510 0x562daf0c4a44 0x562daf0c5035 0x562daf0c9c72 0x562daf0b51f5 0x562daee3a603 0x562daec07a1a 0x562daec0b313 0x562daeb8e159 0x1466a4ac2ec5 0x562daebf2665
 ----- BEGIN BACKTRACE -----
 {"backtrace":[{"b":"562DAE212000","o":"22C0DE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562DAE212000","o":"22BFFF9"},{"b":"562DAE212000","o":"22C04DD"},{"b":"1466A4E67000","o":"10340"},{"b":"1466A4AA1000","o":"36CC9","s":"gsignal"},{"b":"1466A4AA1000","o":"3A0D8","s":"abort"},{"b":"562DAE212000","o":"97A792","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"562DAE212000","o":"EDF510","s":"_ZN5mongo7LogFile17synchronousAppendEPKvm"},{"b":"562DAE212000","o":"EB2A44","s":"_ZN5mongo3dur20_preallocateIsFasterEv"},{"b":"562DAE212000","o":"EB3035","s":"_ZN5mongo3dur19preallocateIsFasterEv"},{"b":"562DAE212000","o":"EB7C72","s":"_ZN5mongo3dur16preallocateFilesEv"},{"b":"562DAE212000","o":"EA31F5","s":"_ZN5mongo3dur7startupEPNS_11ClockSourceEl"},{"b":"562DAE212000","o":"C28603","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"562DAE212000","o":"9F5A1A"},{"b":"562DAE212000","o":"9F9313","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"562DAE212000","o":"97C159","s":"main"},{"b":"1466A4AA1000","o":"21EC5","s":"__libc_start_main"},{"b":"562DAE212000","o":"9E0665"}],"processInfo":{ "mongodbVersion" : "3.6.17-linux-splunk-v4", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.19.107-Unraid", "version" : "#1 SMP Thu Mar 5 13:55:57 PST 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "562DAE212000", "elfType" : 3 }, { "b" : "7FFDFEBB7000", "elfType" : 3 }, { "b" : "1466A5A7A000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3 }, { "b" : "1466A5797000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "1466A5E39000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "1466A5593000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "1466A538B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "1466A5085000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "1466A4E67000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "1466A4AA1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "1466A5C95000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "1466A5E1A000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x562db04d2de1]
  mongod(+0x22BFFF9) [0x562db04d1ff9]
  mongod(+0x22C04DD) [0x562db04d24dd]
  libpthread.so.0(+0x10340) [0x1466a4e77340]
  libc.so.6(gsignal+0x39) [0x1466a4ad7cc9]
  libc.so.6(abort+0x148) [0x1466a4adb0d8]
  mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x562daeb8c792]
  mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x250) [0x562daf0f1510]
  mongod(_ZN5mongo3dur20_preallocateIsFasterEv+0x184) [0x562daf0c4a44]
  mongod(_ZN5mongo3dur19preallocateIsFasterEv+0x35) [0x562daf0c5035]
  mongod(_ZN5mongo3dur16preallocateFilesEv+0x662) [0x562daf0c9c72]
  mongod(_ZN5mongo3dur7startupEPNS_11ClockSourceEl+0x65) [0x562daf0b51f5]
  mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x273) [0x562daee3a603]
  mongod(+0x9F5A1A) [0x562daec07a1a]
  mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x873) [0x562daec0b313]
  mongod(main+0x9) [0x562daeb8e159]
  libc.so.6(__libc_start_main+0xF5) [0x1466a4ac2ec5]
  mongod(+0x9E0665) [0x562daebf2665]
 -----  END BACKTRACE  -----

I've already tried the following:

• created a new server.pem (even though it wasn't expired)
• ensure enough disk space is available
• splunk clean kvstore --all
• chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
• chown -R user:group /opt/splunk/

This is on a single splunk instance and I'm not actively using the kvstore for any of my apps so completely starting fresh with it seemed like the best choice, but even after running clean I'm still getting the same errors.

[venkatasri]

Hi @merrelr 

Could be a possible storage issue - https://jira.mongodb.org/browse/SERVER-9439

This indicates a possible problem with the underlying file system or disk, which prevented Mongo from committing a write. Errno 5 is EIO, which normally indicates bad blocks, transient or permanent file system errors. It is not typically used to indicate lack of free space, but I wouldn't rule it out, either. If this is an EBS or other network backed instance, it could also indicate network problems.

---

Hope it helps!