After from 7.2.1 to 8.6.1 I'm getting the following errors in mongo.log.     2021-10-16T15:01:30.798Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release. 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] MongoDB starting : pid=2763 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=7d49a1b4a62a 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] db version v3.6.17-linux-splunk-v4 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] git version: 226949cc252af265483afbf859b446590b09b098 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2y-fips 16 Feb 2021 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] allocator: tcmalloc 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] modules: none 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] build environment: 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] distarch: x86_64 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] target_arch: x86_64 2021-10-16T15:01:30.840Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "BD6F6380-BE84-4A28-A93F-359500FA793C" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } } 2021-10-16T15:01:30.858Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal 2021-10-16T15:01:30.860Z I JOURNAL [initandlisten] recover : no journal files present, no recovery needed 2021-10-16T15:01:30.863Z I CONTROL [initandlisten] LogFile::synchronousAppend failed with 8192 bytes unwritten out of 8192 bytes; b=0x562db354c000 Bad address 2021-10-16T15:01:30.863Z F - [initandlisten] Fatal Assertion 13515 at src/mongo/db/storage/mmap_v1/logfile.cpp 250 2021-10-16T15:01:30.863Z F - [initandlisten] ***aborting after fassert() failure 2021-10-16T15:01:30.880Z F - [initandlisten] Got signal: 6 (Aborted). 0x562db04d2de1 0x562db04d1ff9 0x562db04d24dd 0x1466a4e77340 0x1466a4ad7cc9 0x1466a4adb0d8 0x562daeb8c792 0x562daf0f1510 0x562daf0c4a44 0x562daf0c5035 0x562daf0c9c72 0x562daf0b51f5 0x562daee3a603 0x562daec07a1a 0x562daec0b313 0x562daeb8e159 0x1466a4ac2ec5 0x562daebf2665 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"562DAE212000","o":"22C0DE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562DAE212000","o":"22BFFF9"},{"b":"562DAE212000","o":"22C04DD"},{"b":"1466A4E67000","o":"10340"},{"b":"1466A4AA1000","o":"36CC9","s":"gsignal"},{"b":"1466A4AA1000","o":"3A0D8","s":"abort"},{"b":"562DAE212000","o":"97A792","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"562DAE212000","o":"EDF510","s":"_ZN5mongo7LogFile17synchronousAppendEPKvm"},{"b":"562DAE212000","o":"EB2A44","s":"_ZN5mongo3dur20_preallocateIsFasterEv"},{"b":"562DAE212000","o":"EB3035","s":"_ZN5mongo3dur19preallocateIsFasterEv"},{"b":"562DAE212000","o":"EB7C72","s":"_ZN5mongo3dur16preallocateFilesEv"},{"b":"562DAE212000","o":"EA31F5","s":"_ZN5mongo3dur7startupEPNS_11ClockSourceEl"},{"b":"562DAE212000","o":"C28603","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"562DAE212000","o":"9F5A1A"},{"b":"562DAE212000","o":"9F9313","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"562DAE212000","o":"97C159","s":"main"},{"b":"1466A4AA1000","o":"21EC5","s":"__libc_start_main"},{"b":"562DAE212000","o":"9E0665"}],"processInfo":{ "mongodbVersion" : "3.6.17-linux-splunk-v4", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.19.107-Unraid", "version" : "#1 SMP Thu Mar 5 13:55:57 PST 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "562DAE212000", "elfType" : 3 }, { "b" : "7FFDFEBB7000", "elfType" : 3 }, { "b" : "1466A5A7A000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3 }, { "b" : "1466A5797000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "1466A5E39000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "1466A5593000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "1466A538B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "1466A5085000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "1466A4E67000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "1466A4AA1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "1466A5C95000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "1466A5E1A000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x562db04d2de1] mongod(+0x22BFFF9) [0x562db04d1ff9] mongod(+0x22C04DD) [0x562db04d24dd] libpthread.so.0(+0x10340) [0x1466a4e77340] libc.so.6(gsignal+0x39) [0x1466a4ad7cc9] libc.so.6(abort+0x148) [0x1466a4adb0d8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x562daeb8c792] mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x250) [0x562daf0f1510] mongod(_ZN5mongo3dur20_preallocateIsFasterEv+0x184) [0x562daf0c4a44] mongod(_ZN5mongo3dur19preallocateIsFasterEv+0x35) [0x562daf0c5035] mongod(_ZN5mongo3dur16preallocateFilesEv+0x662) [0x562daf0c9c72] mongod(_ZN5mongo3dur7startupEPNS_11ClockSourceEl+0x65) [0x562daf0b51f5] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x273) [0x562daee3a603] mongod(+0x9F5A1A) [0x562daec07a1a] mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x873) [0x562daec0b313] mongod(main+0x9) [0x562daeb8e159] libc.so.6(__libc_start_main+0xF5) [0x1466a4ac2ec5] mongod(+0x9E0665) [0x562daebf2665] ----- END BACKTRACE -----     I've already tried the following: created a new server.pem (even though it wasn't expired) ensure enough disk space is available splunk clean kvstore --all chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key chown -R user:group /opt/splunk/   This is on a single splunk instance and I'm not actively using the kvstore for any of my apps so completely starting fresh with it seemed like the best choice, but even after running clean I'm still getting the same errors. ... View more

I'm working on cleaning up permissions for knowledge objects on our search head cluster. I noticed that if I create new knowledge objects and share them with the app their settings in local.meta don't have a line for 'access ='.   Default app permissions example   [savedsearches/test-perms] export = none owner = admin version = 8.1.1 modtime = 1610106340.325106000   After manually changing permissions (even back to the default)   [savedsearches/test-perms] access = read : [ admin ], write : [ admin ] export = none owner = admin version = 8.1.1 modtime = 1610106459.558993000   but no matter what I do to objects that have had permissions set I can't seem to get them to exist without this line and just accept app default permissions.  Does anyone know a safe way to reset a knowledge objects permissions to app defaults without directly modifying local.meta on the sh-cluster members? ... View more

I've setup HEC on a heavy forwarder to gather logs through HEC for Ansible Tower. Logs are rolling in, but I can't seem to get props/transforms setup correctly to rename the hostname from IP to text. props.conf [host::$ipaddress] TRANSFORMS-$hostname_rename = host_rename_$hostname transforms.conf [host_rename_$hostname] REGEX = (.*) DEST_KEY = MetaData:Host FORMAT = host::$hostname I've applied these setting to both the HF and to my Indexer cluster and neither place renames the hostname from IP address to text. Is there something special with HEC or HF that's preventing these changes from taking place? ... View more

I'm looking for a way to report/alert anytime a new forwarder is added to my deployment server. I've tried searching on internal with the following search, but this isn't unique to when a forwarder is initially added to our environment. index=_internal sourcetype=splunkd component=PubSubSvr host=<deploymentServer> ... View more