Same issue here on an Ubuntu 22.04 LTS VM, tried to upgrade on test environment from 9.3.2 to 9.4.0 and mongodb fail to start. Below last mongod_upgrade.log lines 2025-01-27T08:38:58.160Z INFO [mon...
See more...
Same issue here on an Ubuntu 22.04 LTS VM, tried to upgrade on test environment from 9.3.2 to 9.4.0 and mongodb fail to start. Below last mongod_upgrade.log lines 2025-01-27T08:38:58.160Z INFO [mongod_upgrade] All initialized 2025-01-27T08:38:58.160Z INFO [mongod_upgrade] Upgrading to 4.4 2025-01-27T08:38:58.161Z INFO [mongod_upgrade] Waiting if primary 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(83), "optime": Document({"ts": Timestamp { time: 1737967138, increment: 2 }, "t": Int64(57)}), "optimeDate": DateTime(2025-01-27 8:38:58.0 +00:00:00), "syncingTo": String(""), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("could not find member to sync from"), "electionTime": Timestamp { time: 1737967059, increment: 1 }, "electionDate": DateTime(2025-01-27 8:37:39.0 +00:00:00), "configVersion": Int32(1), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:38:58.162Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:38:58.163Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:38:58.163Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:38:58.164Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 4.4 2025-01-27T08:38:58.164Z INFO [mongod_upgrade::commands] In update for 4.4 2025-01-27T08:38:59.046Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:38:59.822Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(8), source: None } 2025-01-27T08:39:05.832Z INFO [mongod_upgrade::commands] Checking if mongod is online 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] mongod is offline 2025-01-27T08:39:35.834Z INFO [mongod_upgrade::commands] Shutdown output: Document({}) 2025-01-27T08:39:37.263Z INFO [mongod_upgrade::commands] UPGRADE_TO_4.4_SUCCESSFUL 2025-01-27T08:39:39.263Z INFO [mongod_upgrade::commands] Attempting to update status 2025-01-27T08:39:39.265Z INFO [mongod_upgrade::commands] Status updated successfully 2025-01-27T08:39:39.271Z INFO [mongod_upgrade] Waiting for other nodes in replica set to upgrade 2025-01-27T08:39:39.272Z DEBUG [mongod_upgrade::conditions] Upgraded count: 1 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] All upgraded to 4.4, proceeding. 2025-01-27T08:39:39.272Z INFO [mongod_upgrade] Setting new FCV Version: 4.4 2025-01-27T08:39:39.284Z INFO [mongod_upgrade] FCV change successful: () 2025-01-27T08:39:54.284Z INFO [mongod_upgrade] Upgrading to 5.0 2025-01-27T08:39:54.286Z INFO [mongod_upgrade] Waiting if primary 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Replication status doc for node: Document({"_id": Int32(0), "name": String("127.0.0.1:8191"), "health": Double(1.0), "state": Int32(1), "stateStr": String("PRIMARY"), "uptime": Int32(19), "optime": Document({"ts": Timestamp { time: 1737967179, increment: 4 }, "t": Int64(58)}), "optimeDate": DateTime(2025-01-27 8:39:39.0 +00:00:00), "lastAppliedWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "lastDurableWallTime": DateTime(2025-01-27 8:39:39.278 +00:00:00), "syncSourceHost": String(""), "syncSourceId": Int32(-1), "infoMessage": String("Could not find member to sync from"), "electionTime": Timestamp { time: 1737967177, increment: 1 }, "electionDate": DateTime(2025-01-27 8:39:37.0 +00:00:00), "configVersion": Int32(2), "configTerm": Int32(58), "self": Boolean(true), "lastHeartbeatMessage": String("")}) 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from replSetGetStatus: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Hostname from preflight: "127.0.0.1:8191" 2025-01-27T08:39:54.287Z DEBUG [mongod_upgrade::conditions] Upgraded count: 0 2025-01-27T08:39:54.287Z INFO [mongod_upgrade] Getting lock 2025-01-27T08:39:54.288Z DEBUG [mongod_upgrade::conditions] Upserting lock 2025-01-27T08:39:54.288Z INFO [mongod_upgrade::conditions] locked 2025-01-27T08:39:54.288Z INFO [mongod_upgrade] Got lock: true 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] Updating 127.0.0.1:8191 to 5.0 2025-01-27T08:39:54.289Z INFO [mongod_upgrade::commands] In update for 5.0 2025-01-27T08:39:54.555Z INFO [mongod_upgrade::commands] Shutting down the database 2025-01-27T08:39:55.409Z WARN [mongod_upgrade::commands] Attempting with force:true due to shutdown failure: Error { kind: Io(Kind(UnexpectedEof)), labels: {}, wire_version: Some(9), source: None } and below mongod.log after Splunk start/restart 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-01-27T15:56:52.142Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1075M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2025-01-27T15:56:53.003Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:3955][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.027Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:27009][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.049Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:49077][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.080Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:80951][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.096Z E STORAGE [initandlisten] WiredTiger error (-31802) [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1737993413:96579][338955:0x7f36dba58b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2025-01-27T15:56:53.103Z F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error 2025-01-27T15:56:53.103Z F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 928 2025-01-27T15:56:53.103Z F - [initandlisten] \n\n***aborting after fassert() failure\n\n we have avx and avx2 CPU Flags $ lscpu | grep -i avx Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities it seems that mongodb is in stuck state and did not completed the upgrade correctly. Any suggestion how to resolve? Regards