In our test environment we downgraded to 9.3.2 and KV Store was not started with the same error message in the log file, maybe the mongodb was corrupted. As reported in the Splunk docs here: https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/MigrateKVstore the mongodb server need to be on 4.2.x version: You must upgrade to server version 4.2.x before upgrading to Splunk Enterprise 9.4.x or higher. For instructions and information about updating to KV store server version 4.2.x in Splunk Enterprise versions 9.0.x through 9.3.x, see Migrate the KV store storage engine in the Splunk Enterprise 9.3.0 documentation.   so, to check, it is strongly suggested to see the following Splunk guide: https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/MigrateKVstore After that, we stopped Splunk and issued a splunk clean kvstore --local after restarting Splunk everything was back working. We upgraded again to 9.4.0, after some seconds it starts to upgrade mongodb from 4.2 to 7.0 through 4.4, 5.0 and 6.0 with message in GUI that KV store is updating, we need to wait until update its finished. After some minutes mongodb was been successfully updated with the following message in the Splunk GUI: and the Splunk version: it is strongly suggested to tailing the $SPLUNK_HOME/var/log/splunk/mongodb_upgrade.log and do not operate on Splunk until update is finished. below the mongodb_upgrade.log 2025-01-28T08:36:25.567Z INFO [mongod_upgrade] Mongod Upgrader Logs 2025-01-28T08:36:25.568Z DEBUG [mongod_upgrade] mongod_upgrade arguments: Args { verbose: Verbosity { verbose: 1, quiet: 0, phantom: PhantomData<clap_verbosity::InfoLevel> }, uri: "mongodb://__system@127.0.0.1:8191/?replicaSet=B99AB2AA-EE93-405A-95CD-89EAC0FCA551&retryWrites=true&authSource=local&ssl=true&connectTimeoutMS=10000&socketTimeoutMS=300000&readPreference=nearest&readPreferenceTags=instance:B99AB2AA-EE93-405A-95CD-89EAC0FCA551&readPreferenceTags=all:all&tlsAllowInvalidCertificates=true", nodes: 1, local_uri: "mongodb://__system@127.0.0.1:8191/?w=majority&journal=true&retryWrites=true&authSource=local&ssl=true&connectTimeoutMS=10000&socketTimeoutMS=300000&tlsAllowInvalidCertificates=true&directConnection=true", backup_disabled: false, ld_linker: "lib/ld-2.26.so", data_directory: "/opt/splunk/var/lib/splunk/kvstore/mongo", backup_path: "/opt/splunk/var/lib/splunk/kvstore/mongo_backup", shadow_mount_dir: "C:\\mongo_shadow\\", backup_volume: "NOT_SPECIFIED", logpath: "/opt/splunk/var/log/splunk/mongod.log", keep_metadata: false, drop_metadata: false, pre_drop_metadata: false, keep_backups: true, metadata_database: "migration_metadata", metadata_collection: "migration_metadata", rsync_retries: 5, max_start_retries: 10, shutdown_mongod: true, rsync_path: "/opt/splunk/bin/rsync", keyfile_path: Some("/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key"), max_command_time_ms: 60000, time_duration_block_s: 1, polling_interval_ms: 100, polling_max_wait_ms: 26460000, polling_version_max_wait_ms: 4800000, max_retries: 4, health_check_max_retries: 60, use_ld: false, mongod_args: ["--dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo", "--storageEngine=wiredTiger", "--wiredTigerCacheSizeGB=1.050000", "--port=8191", "--timeStampFormat=iso8601-utc", "--oplogSize=200", "--keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key", "--setParameter=enableLocalhostAuthBypass=0", "--setParameter=oplogFetcherSteadyStateMaxFetcherRestarts=0", "--replSet=B99AB2AA-EE93-405A-95CD-89EAC0FCA551", "--bind_ip=0.0.0.0", "--sslCAFile=/opt/splunk/etc/auth/cacert.pem", "--tlsAllowConnectionsWithoutCertificates", "--sslMode=requireSSL", "--sslAllowInvalidHostnames", "--sslPEMKeyFile=/opt/splunk/etc/auth/server.pem", "--sslPEMKeyPassword=password", "--tlsDisabledProtocols=noTLS1_0,noTLS1_1", "--sslCipherConfig=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256", "--nounixsocket", "--noscripting"] } 2025-01-28T08:36:25.568Z INFO [mongod_upgrade] Executing Preflight Checks 2025-01-28T08:36:25.568Z DEBUG [mongod_upgrade] client_options_primary: ClientOptions { hosts: [Tcp { host: "127.0.0.1", port: Some(8191) }], app_name: None, compressors: None, connect_timeout: Some(10s), credential: Some(Credential("REDACTED")), direct_connection: None, driver_info: None, heartbeat_freq: None, load_balanced: None, local_threshold: None, max_idle_time: None, max_pool_size: None, min_pool_size: None, max_connecting: None, read_concern: None, repl_set_name: Some("B99AB2AA-EE93-405A-95CD-89EAC0FCA551"), retry_reads: None, retry_writes: Some(true), selection_criteria: Some(ReadPreference(Nearest { options: ReadPreferenceOptions { tag_sets: Some([{"instance": "B99AB2AA-EE93-405A-95CD-89EAC0FCA551"}, {"all": "all"}]), max_staleness: None, hedge: None } })), server_api: None, server_selection_timeout: None, default_database: None, tls: Some(Enabled(TlsOptions { allow_invalid_certificates: Some(true), ca_file_path: None, cert_key_file_path: None, allow_invalid_hostnames: None })), write_concern: None, srv_max_hosts: None } 2025-01-28T08:36:25.587Z INFO [mongod_upgrade] Checking intial FCV 2025-01-28T08:36:25.587Z INFO [mongod_upgrade] Feature Compatibility Version is: 4.2 2025-01-28T08:36:25.587Z DEBUG [mongod_upgrade] Hostname set to "127.0.0.1:8191" 2025-01-28T08:36:25.588Z INFO [mongod_upgrade] Preflight completed successfully 2025-01-28T08:36:25.589Z INFO [mongod_upgrade] Executing backup before upgrade 2025-01-28T08:36:25.613Z DEBUG [mongod_upgrade] Backup update doc: Document({"$set": Document({"backup.location": String("/opt/splunk/var/lib/splunk/kvstore/mongo_backup"), "backup.start": DateTime(2025-01-28 8:36:25.613 +00:00:00), "backup.phase1_start": DateTime(2025-01-28 8:36:25.613 +00:00:00)})}) 2025-01-28T08:36:25.662Z DEBUG [mongod_upgrade] Backup update result: UpdateResult { matched_count: 0, modified_count: 0, upserted_id: Some(String("BACKUP_127.0.0.1:8191")) } 2025-01-28T08:36:26.426Z DEBUG [mongod_upgrade] Rsync returned successfully. 2025-01-28T08:36:26.426Z DEBUG [mongod_upgrade] Backup update doc: Document({"$set": Document({"backup.phase1_end": DateTime(2025-01-28 8:36:26.426 +00:00:00), "backup.phase2_start": DateTime(2025-01-28 8:36:26.426 +00:00:00)})}) 2025-01-28T08:36:26.428Z DEBUG [mongod_upgrade] Backup update result: UpdateResult { matched_count: 1, modified_count: 1, upserted_id: None } 2025-01-28T08:36:26.429Z DEBUG [mongod_upgrade::conditions] "phase1" complete count: 1 2025-01-28T08:36:26.433Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(61), "optime": Document({"ts": Timestamp { time: 1738053386, increment: 1 }, "t": Int64(2)}), "optimeDate": DateTime(2025-01-28 8:36:26.0 +00:00:00), "syncingTo": String(""), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("could not find member to sync from"), "electionTime": Timestamp { time: 1738053327, increment: 1 }, "electionDate": DateTime(2025-01-28 8:35:27.0 +00:00:00), "configVersion": Int32(1), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-28T08:36:26.434Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-28T08:36:26.434Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-28T08:36:26.434Z INFO [mongod_upgrade] Node identified as Primary, issuing fsyncLock and pausing writes to node 2025-01-28T08:36:26.578Z INFO [mongod_upgrade] Document({"info": String("now locked against writes, use db.fsyncUnlock() to unlock"), "lockCount": Int64(1), "seeAlso": String("http://dochub.mongodb.org/core/fsynccommand"), "ok": Double(1.0), "$clusterTime": Document({"clusterTime": Timestamp { time: 1738053386, increment: 1 }, "signature": Document({"hash": Binary { subtype: Generic, bytes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }, "keyId": Int64(0)})}), "operationTime": Timestamp { time: 1738053386, increment: 1 }}) 2025-01-28T08:36:26.578Z INFO [mongod_upgrade] Waiting for replication lag to be 0 on all secondary nodes 2025-01-28T08:36:27.759Z INFO [mongod_upgrade] unpausing writes to node 2025-01-28T08:36:27.764Z INFO [mongod_upgrade] Document({"info": String("fsyncUnlock completed"), "lockCount": Int64(0), "ok": Double(1.0), "$clusterTime": Document({"clusterTime": Timestamp { time: 1738053386, increment: 1 }, "signature": Document({"hash": Binary { subtype: Generic, bytes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }, "keyId": Int64(0)})}), "operationTime": Timestamp { time: 1738053386, increment: 1 }}) 2025-01-28T08:36:27.764Z DEBUG [mongod_upgrade] Second rsync returned successfully. 2025-01-28T08:36:27.764Z DEBUG [mongod_upgrade] Backup update doc: Document({"$set": Document({"backup.phase2_end": DateTime(2025-01-28 8:36:27.764 +00:00:00), "backup.end": DateTime(2025-01-28 8:36:27.764 +00:00:00)})}) 2025-01-28T08:36:27.765Z DEBUG [mongod_upgrade] Backup update result: UpdateResult { matched_count: 1, modified_count: 1, upserted_id: None } 2025-01-28T08:36:27.766Z DEBUG [mongod_upgrade::conditions] "phase2" complete count: 1 2025-01-28T08:36:27.766Z INFO [mongod_upgrade] Backup completed successfully 2025-01-28T08:36:27.766Z INFO [mongod_upgrade] Starting rolling update 2025-01-28T08:36:27.785Z INFO [mongod_upgrade::commands] Init results: InsertOneResult { inserted_id: String("127.0.0.1:8191") } 2025-01-28T08:36:27.785Z INFO [mongod_upgrade] Waiting for initialization 2025-01-28T08:36:27.786Z DEBUG [mongod_upgrade::conditions] Init count: 1 2025-01-28T08:36:27.786Z INFO [mongod_upgrade] All initialized 2025-01-28T08:36:27.786Z INFO [mongod_upgrade] Upgrading to 4.4 2025-01-28T08:36:27.787Z INFO [mongod_upgrade] Waiting if primary 2025-01-28T08:36:27.788Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(62), "optime": Document({"ts": Timestamp { time: 1738053387, increment: 2 }, "t": Int64(2)}), "optimeDate": DateTime(2025-01-28 8:36:27.0 +00:00:00), "syncingTo": String(""), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("could not find member to sync from"), "electionTime": Timestamp { time: 1738053327, increment: 1 }, "electionDate": DateTime(2025-01-28 8:35:27.0 +00:00:00), "configVersion": Int32(1), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-28T08:36:27.788Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-28T08:36:27.788Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-28T08:36:27.788Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-28T08:36:27.788Z INFO [mongod_upgrade] Getting lock 2025-01-28T08:36:27.789Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-28T08:36:27.790Z INFO [mongod_upgrade::conditions] locked 2025-01-28T08:36:27.790Z INFO [mongod_upgrade] Got lock: true 2025-01-28T08:36:27.790Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 4.4 2025-01-28T08:36:27.790Z INFO [mongod_upgrade::commands] In update for 4.4 2025-01-28T08:36:29.202Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-28T08:36:29.736Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(8), source: None } 2025-01-28T08:36:35.754Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-28T08:37:05.755Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-28T08:37:05.755Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-28T08:37:07.813Z INFO [mongod_upgrade::commands] UPGRADE_TO_4.4_SUCCESSFUL 2025-01-28T08:37:09.813Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-28T08:37:09.817Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-28T08:37:09.823Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-28T08:37:09.824Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-28T08:37:09.824Z INFO [mongod_upgrade] All upgraded to 4.4, proceeding. 2025-01-28T08:37:09.824Z INFO [mongod_upgrade] Setting new FCV Version: 4.4 2025-01-28T08:37:09.838Z INFO [mongod_upgrade] FCV change successful: () 2025-01-28T08:37:24.838Z INFO [mongod_upgrade] Upgrading to 5.0 2025-01-28T08:37:24.840Z INFO [mongod_upgrade] Waiting if primary 2025-01-28T08:37:24.841Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(18), "optime": Document({"ts": Timestamp { time: 1738053429, increment: 4 }, "t": Int64(3)}), "optimeDate": DateTime(2025-01-28 8:37:09.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-28 8:37:09.834 +00:00:00), "lastDurableWallTime": DateTime(2025-01-28 8:37:09.834 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1738053427, increment: 1 }, "electionDate": DateTime(2025-01-28 8:37:07.0 +00:00:00), "configVersion": Int32(2), "configTerm": Int32(3), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-28T08:37:24.841Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-28T08:37:24.841Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-28T08:37:24.842Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-28T08:37:24.842Z INFO [mongod_upgrade] Getting lock 2025-01-28T08:37:24.842Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-28T08:37:24.843Z INFO [mongod_upgrade::conditions] locked 2025-01-28T08:37:24.843Z INFO [mongod_upgrade] Got lock: true 2025-01-28T08:37:24.843Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 5.0 2025-01-28T08:37:24.843Z INFO [mongod_upgrade::commands] In update for 5.0 2025-01-28T08:37:26.825Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-28T08:37:27.994Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(9), source: None } 2025-01-28T08:37:34.004Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-28T08:38:04.006Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-28T08:38:04.006Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-28T08:38:06.710Z INFO [mongod_upgrade::commands] UPGRADE_TO_5.0_SUCCESSFUL 2025-01-28T08:38:08.710Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-28T08:38:08.717Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-28T08:38:08.725Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-28T08:38:08.732Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-28T08:38:08.732Z INFO [mongod_upgrade] All upgraded to 5.0, proceeding. 2025-01-28T08:38:08.732Z INFO [mongod_upgrade] Setting new FCV Version: 5.0 2025-01-28T08:38:08.748Z INFO [mongod_upgrade] FCV change successful: () 2025-01-28T08:38:23.748Z INFO [mongod_upgrade] Upgrading to 6.0 2025-01-28T08:38:23.751Z INFO [mongod_upgrade] Waiting if primary 2025-01-28T08:38:23.752Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(19), "optime": Document({"ts": Timestamp { time: 1738053488, increment: 5 }, "t": Int64(4)}), "optimeDate": DateTime(2025-01-28 8:38:08.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-28 8:38:08.745 +00:00:00), "lastDurableWallTime": DateTime(2025-01-28 8:38:08.745 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1738053486, increment: 1 }, "electionDate": DateTime(2025-01-28 8:38:06.0 +00:00:00), "configVersion": Int32(3), "configTerm": Int32(4), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-28T08:38:23.752Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-28T08:38:23.752Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-28T08:38:23.752Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-28T08:38:23.752Z INFO [mongod_upgrade] Getting lock 2025-01-28T08:38:23.753Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-28T08:38:23.757Z INFO [mongod_upgrade::conditions] locked 2025-01-28T08:38:23.757Z INFO [mongod_upgrade] Got lock: true 2025-01-28T08:38:23.758Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 6.0 2025-01-28T08:38:23.758Z INFO [mongod_upgrade::commands] In update for 6.0 2025-01-28T08:38:25.544Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-28T08:38:25.845Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(13), source: None } 2025-01-28T08:38:31.854Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-28T08:39:01.856Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-28T08:39:01.856Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-28T08:39:03.281Z INFO [mongod_upgrade::commands] UPGRADE_TO_6.0_SUCCESSFUL 2025-01-28T08:39:05.281Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-28T08:39:05.285Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-28T08:39:05.297Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-28T08:39:05.299Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-28T08:39:05.299Z INFO [mongod_upgrade] All upgraded to 6.0, proceeding. 2025-01-28T08:39:05.299Z INFO [mongod_upgrade] Setting new FCV Version: 6.0 2025-01-28T08:39:05.460Z INFO [mongod_upgrade] FCV change successful: () 2025-01-28T08:39:20.460Z INFO [mongod_upgrade] Upgrading to 7.0 2025-01-28T08:39:20.462Z INFO [mongod_upgrade] Waiting if primary 2025-01-28T08:39:20.464Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(19), "optime": Document({"ts": Timestamp { time: 1738053545, increment: 10 }, "t": Int64(5)}), "optimeDate": DateTime(2025-01-28 8:39:05.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-28 8:39:05.451 +00:00:00), "lastDurableWallTime": DateTime(2025-01-28 8:39:05.451 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1738053543, increment: 1 }, "electionDate": DateTime(2025-01-28 8:39:03.0 +00:00:00), "configVersion": Int32(3), "configTerm": Int32(5), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-28T08:39:20.464Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-28T08:39:20.464Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-28T08:39:20.465Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-28T08:39:20.465Z INFO [mongod_upgrade] Getting lock 2025-01-28T08:39:20.466Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-28T08:39:20.469Z INFO [mongod_upgrade::conditions] locked 2025-01-28T08:39:20.469Z INFO [mongod_upgrade] Got lock: true 2025-01-28T08:39:20.470Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 7.0 2025-01-28T08:39:20.470Z INFO [mongod_upgrade::commands] In update for 7.0 2025-01-28T08:39:21.724Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-28T08:39:22.519Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(17), source: None } 2025-01-28T08:39:28.529Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-28T08:39:58.531Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-28T08:39:58.531Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-28T08:40:00.234Z INFO [mongod_upgrade::commands] UPGRADE_TO_7.0_SUCCESSFUL 2025-01-28T08:40:02.234Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-28T08:40:02.240Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-28T08:40:02.253Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-28T08:40:02.258Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-28T08:40:02.258Z INFO [mongod_upgrade] All upgraded to 7.0, proceeding. 2025-01-28T08:40:02.259Z INFO [mongod_upgrade] Setting new FCV Version: 7.0 2025-01-28T08:40:02.281Z INFO [mongod_upgrade] FCV change successful: () 2025-01-28T08:40:17.281Z INFO [mongod_upgrade] Upgrades completed 2025-01-28T08:40:17.287Z INFO [mongod_upgrade] Waiting for completion 2025-01-28T08:40:17.289Z DEBUG [mongod_upgrade::conditions] Completed count: 1 2025-01-28T08:40:17.289Z DEBUG [mongod_upgrade::conditions] Hostname count: 1 2025-01-28T08:40:17.289Z INFO [mongod_upgrade] All completed 2025-01-28T08:40:17.394Z INFO [mongod_upgrade] Dropped migration metadata database. 2025-01-28T08:40:17.403Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-28T08:40:18.363Z WARN [mongod_upgrade] mongod failed to shut down before exiting: Kind: I/O error: unexpected end of file, labels: {}  and below the correct mongodb version splunk show kvstore-status --verbose | grep -i serverversion serverVersion : 7.0.14 Since ours was a test server, and other than the default Splunk server we don't use anything that uses the KV Store, we didn't restore the backed up KV Store. Splunk does it automatically when upgrading the version, but it is recommended to do it manually and then restore the KV Store in case something goes wrong. Furthermore, it is never recommended to upgrade Splunk to version .0, but at least wait for the next versions, for example .3 because many bugs are solved that are often present in the .0 version (eg: 9.4.3 version). ... View more

Same issue here on a VM with Ubuntu 22.04 LTS, after upgrade from Splunk 9.3.2 to Splunk 9.4.0, mongodb fail to start. The "avx" and "avx2" vCPU flags are ok: $ lscpu | grep -i avx Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities below latest mongod_upgrade.log lines 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:38:58.163Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:38:58.164Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 4.4 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] In update for 4.4 2025-01-27T08:38:59.046Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:38:59.822Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(8), source: None } 2025-01-27T08:39:05.832Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-27T08:39:37.263Z INFO [mongod_upgrade::commands] UPGRADE_TO_4.4_SUCCESSFUL 2025-01-27T08:39:39.263Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-27T08:39:39.265Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-27T08:39:39.271Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-27T08:39:39.272Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] All upgraded to 4.4, proceeding. 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] Setting new FCV Version: 4.4 2025-01-27T08:39:39.284Z INFO [mongod_upgrade] FCV change successful: () 2025-01-27T08:39:54.284Z INFO [mongod_upgrade] Upgrading to 5.0 2025-01-27T08:39:54.286Z INFO [mongod_upgrade] Waiting if primary 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(19), "optime": Document({"ts": Timestamp { time: 1737967179, increment: 4 }, "t": Int64(58)}), "optimeDate": DateTime(2025-01-27 8:39:39.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "lastDurableWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1737967177, increment: 1 }, "electionDate": DateTime(2025-01-27 8:39:37.0 +00:00:00), "configVersion": Int32(2), "configTerm": Int32(58), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:39:54.287Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:39:54.288Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:39:54.288Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:39:54.288Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 5.0 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] In update for 5.0 2025-01-27T08:39:54.555Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:39:55.409Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(9), source: None }  and below the mongod.log error after a splunk start/restart 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1075M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2025-01-27T15:56:53.003Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.027Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.049Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.080Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.096Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2025-01-27T15:56:53.103Z F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 928 2025-01-27T15:56:53.103Z F - [initandlisten] \n\n***aborting after fassert() failure\n\n  it seems that was stuck on 5.0 mongodb upgrade after 9.4.0 Splunk upgrade. Any suggestion how to resolve the issue? Regards. ... View more

Same issue here on an Ubuntu 22.04 LTS VM, tried to upgrade on test environment from 9.3.2 to 9.4.0 and mongodb fail to start. Below last mongod_upgrade.log lines 2025-01-27T08:38:58.160Z INFO [mongod_upgrade] All initialized 2025-01-27T08:38:58.160Z INFO [mongod_upgrade] Upgrading to 4.4 2025-01-27T08:38:58.161Z INFO [mongod_upgrade] Waiting if primary 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(83), "optime": Document({"ts": Timestamp { time: 1737967138, increment: 2 }, "t": Int64(57)}), "optimeDate": DateTime(2025-01-27 8:38:58.0 +00:00:00), "syncingTo": String(""), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("could not find member to sync from"), "electionTime": Timestamp { time: 1737967059, increment: 1 }, "electionDate": DateTime(2025-01-27 8:37:39.0 +00:00:00), "configVersion": Int32(1), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:38:58.163Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:38:58.164Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 4.4 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] In update for 4.4 2025-01-27T08:38:59.046Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:38:59.822Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(8), source: None } 2025-01-27T08:39:05.832Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-27T08:39:37.263Z INFO [mongod_upgrade::commands] UPGRADE_TO_4.4_SUCCESSFUL 2025-01-27T08:39:39.263Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-27T08:39:39.265Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-27T08:39:39.271Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-27T08:39:39.272Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] All upgraded to 4.4, proceeding. 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] Setting new FCV Version: 4.4 2025-01-27T08:39:39.284Z INFO [mongod_upgrade] FCV change successful: () 2025-01-27T08:39:54.284Z INFO [mongod_upgrade] Upgrading to 5.0 2025-01-27T08:39:54.286Z INFO [mongod_upgrade] Waiting if primary 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(19), "optime": Document({"ts": Timestamp { time: 1737967179, increment: 4 }, "t": Int64(58)}), "optimeDate": DateTime(2025-01-27 8:39:39.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "lastDurableWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1737967177, increment: 1 }, "electionDate": DateTime(2025-01-27 8:39:37.0 +00:00:00), "configVersion": Int32(2), "configTerm": Int32(58), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:39:54.287Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:39:54.288Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:39:54.288Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:39:54.288Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 5.0 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] In update for 5.0 2025-01-27T08:39:54.555Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:39:55.409Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(9), source: None } and below mongod.log after Splunk start/restart 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1075M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2025-01-27T15:56:53.003Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.027Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.049Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.080Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.096Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2025-01-27T15:56:53.103Z F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 928 2025-01-27T15:56:53.103Z F - [initandlisten] \n\n***aborting after fassert() failure\n\n we have avx and avx2 CPU Flags $ lscpu | grep -i avx Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities it seems that mongodb is in stuck state and did not completed the upgrade correctly. Any suggestion how to resolve? Regards ... View more

Sorry, I don't need proxy to be configured globally, but I need proxy configuration per App. The proxy configuration mentioned in above url is for splunkd only and doesn't work with Apps. ... View more

The WA works like a charm and fixes the second issue. I also suggest you to add some detailed documentation about App, to better understand all the functionalities. Thank you for quick WA and for this awesome App. ... View more

Same issue for me on single Splunk v6.6.3 instance (SH and Indexer on single Server) with FireEye App for Splunk Enterprise v3.1.1. If I try search in FireEye App, fieldalias doesn't work, so dashboards not populated correctly. When I try search in Splunk default search App, all fieldalias extracting correctly. I've tried setting global permission on App with no chance. Any suggestion? Regards ... View more

Thank you for answer. I'll submit a case to Splunk Support. ... View more

Hi walterk82 and thank you for your answer. I try to explain, I think there's another issue, in TA default/eventtypes.conf 61-62 lines there's configured: [f5_bigip_apm_username_received] search = sourcetype="f5:bigip:apm:syslog" ": Username" above stanza recall authentication dataset action in default/tags.conf 117-123 lines: [eventtype=f5_bigip_apm_username_received] network = enabled communicate = enabled session = enabled authentication = enabled default = enabled web = enabled so when I try to searching for tag=authentication the action field was populated in "Access policy result:" rows only, not in "Username" rows with field values "success" or "failure" that are CIM expected values for Authentication datasets for populate Splunk ITSI or Splunk ES Premium Apps. May know a temporary workaround to put in the TA local/props.conf for extract Username success or failure action as expected? Thank you in advance for any help. ... View more

Ok I did the trick! Forwarder Upgrade with no custom options create Splunk_TA_windows that override my "domain_controller" app under UF path "$SPLUNK_HOME\etc\apps", I ran btool command and output following result: C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog so I've uninstalled UF and reinstalled with "Customize Options" and uncheck all flags under "Windows Event Logs", after installation I've checked that "Splunk_TA_windows" directory that not exists under "$SPLUNK_HOME\etc\apps" then Start SplunkUniversalForwarder Service, Deployment Server pushed configurations to Deployment Client and now I receive logs. Thx all for support!!! ... View more

No override, local directory is empty. ... View more

Ok I'll try to execute above command and paste output. ... View more

I know that events going to new index that not exists, but my trouble is another. I've plain Splunk Enterprise Core, no TA, my issue is particular because before with SUF 5.0.2 to Indexer 6.4.1 Windows Event Log was Forwarded, after "clean" upgrade to SUF 6.4.3 Forwarder stop to send Event Log to Indexer 6.4.1. Before SUF upgrade Event Logs were going to "main" index with no explicit declaration in inputs.conf, so I've tried to configure explicitly the previous used index in inputs.conf and reload serverclasses with no success, and I don't need to create new index because prior upgrade this worked. In Preproduction Environment all works fine and SUF forward events log to Indexer. I hope I explained myself. ... View more