@splunkreal  As I understand from the end of the mongo.log file splunk stops the mongo service , but dont know why   WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","log_id":1000000,"category_id":7,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 1, snapshot max: 1 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 123"}}} {"t":{"$date":"2026-07-28T14:59:00.705Z"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":705846,"thread":"165465:0x7fe1b96b0700","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","log_id":1000000,"category_id":34,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"recovery checkpoint has successfully finished and ran for 11 milliseconds"}}} {"t":{"$date":"2026-07-28T14:59:00.705Z"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":705907,"thread":"165465:0x7fe1b96b0700","session_name":"txn-recover","category":"WT_VERB_RECOVERY","log_id":1493201,"category_id":33,"verbose_level":"INFO","verbose_level_id":0,"msg":"recovery was completed successfully and took 269ms, including 256ms for the log replay, 0ms for the rollback to stable, and 11ms for the checkpoint."}}} {"t":{"$date":"2026-07-28T14:59:00.705Z"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":705925,"thread":"165465:0x7fe1b96b0700","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","log_id":1493201,"category_id":34,"verbose_level":"INFO","verbose_level_id":0,"msg":"recovery was completed successfully and took 269ms, including 256ms for the log replay, 0ms for the rollback to stable, and 11ms for the checkpoint."}}} {"t":{"$date":"2026-07-28T14:59:00.707Z"},"s":"I", "c":"STORAGE", "id":4795904, "ctx":"SignalHandler","msg":"WiredTiger re-opened","attr":{"durationMillis":300}} {"t":{"$date":"2026-07-28T14:59:00.707Z"},"s":"I", "c":"STORAGE", "id":22325, "ctx":"SignalHandler","msg":"Reconfiguring","attr":{"newConfig":"compatibility=(release=10.0)"}} {"t":{"$date":"2026-07-28T14:59:00.717Z"},"s":"I", "c":"STORAGE", "id":4795903, "ctx":"SignalHandler","msg":"Reconfigure complete","attr":{"durationMillis":10}} {"t":{"$date":"2026-07-28T14:59:00.717Z"},"s":"I", "c":"STORAGE", "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,use_timestamp=false,"}} {"t":{"$date":"2026-07-28T14:59:00.718Z"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":718796,"thread":"165465:0x7fe1b96b0700","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","log_id":1000000,"category_id":7,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 2, snapshot max: 2 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 123"}}} {"t":{"$date":"2026-07-28T14:59:00.725Z"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":725401,"thread":"165465:0x7fe1b96b0700","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","log_id":1000000,"category_id":34,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 7 milliseconds"}}} {"t":{"$date":"2026-07-28T14:59:00.725Z"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":{"message":{"ts_sec":1785250740,"ts_usec":725518,"thread":"165465:0x7fe1b96b0700","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","log_id":1493200,"category_id":34,"verbose_level":"INFO","verbose_level_id":0,"msg":"shutdown was completed successfully and took 7ms, including 0ms for the rollback to stable, and 7ms for the checkpoint."}}} {"t":{"$date":"2026-07-28T14:59:00.733Z"},"s":"I", "c":"STORAGE", "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"durationMillis":16}} {"t":{"$date":"2026-07-28T14:59:00.733Z"},"s":"I", "c":"STORAGE", "id":22279, "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."} {"t":{"$date":"2026-07-28T14:59:00.733Z"},"s":"I", "c":"-", "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"} {"t":{"$date":"2026-07-28T14:59:00.733Z"},"s":"I", "c":"FTDC", "id":20626, "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"} {"t":{"$date":"2026-07-28T14:59:00.733Z"},"s":"I", "c":"-", "id":10175800,"ctx":"SignalHandler","msg":"Shutting down the standalone executor"} {"t":{"$date":"2026-07-28T14:59:00.734Z"},"s":"I", "c":"ASIO", "id":22582, "ctx":"StandaloneNetwork","msg":"Killing all outstanding egress activity."} {"t":{"$date":"2026-07-28T14:59:00.734Z"},"s":"I", "c":"CONTROL", "id":20565, "ctx":"SignalHandler","msg":"Now exiting"} {"t":{"$date":"2026-07-28T14:59:00.734Z"},"s":"I", "c":"CONTROL", "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":{"Summary of time elapsed":{"Statistics":{"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"0 ms","Time spent in quiesce mode":"0 ms","Shut down FLE Crud subsystem":"0 ms","Shut down MirrorMaestro":"1 ms","Shut down WaitForMajorityService":"0 ms","Shut down the logical session cache":"0 ms","Shut down the Query Analysis Sampler":"0 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"0 ms","Shut down the thread that aborts expired transactions":"0 ms","Shut down the replica set aware services":"0 ms","Shut down replication":"0 ms","Shut down external state":"0 ms","Shut down replication executor":"0 ms","Join replication executor":"0 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"1 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"0 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"0 ms","Shut down the transport layer":"0 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"0 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"1338 ms","Shut down full-time data capture":"0 ms","Shut down standalone executor":"1 ms","Shut down online certificate status protocol manager":"0 ms","shutdownTask total elapsed time":"1342 ms"}}}} {"t":{"$date":"2026-07-28T14:59:00.734Z"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}} ... View more

I still have issues for some reason the mongo starts but stops after 2mins and the KVstore never starts. Do I need to run any migration ? or i should update the conf files to keep mongo always on ?  ... View more

@burwell  ./splunk start-standalone-upgrade kvstore -dryRun true In handler 'kvstoreupgrade': KVStore not in ready state. Upgrade cannot proceed. Current Status :: startingMissing Mongod Binaries :: /opt/splunk/bin/mongod-6.0; /opt/splunk/bin/mongod-5.0; /opt/splunk/bin/mongod-4.4; Please make sure they are present under :: /opt/splunk/bin before proceeding with upgrade. Upgrade Path = /opt/splunk/bin/mongod_upgrade not found Please make sure upgrade tool binary exists under /opt/splunk/binFailed to check current FCV value prior to KV Store upgrade. Error= An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]. Topology type: ReplicaSetNoPrimary .Please check KVStore Status ... View more

My splunk version now is 10.4.0, I will check kvm store version and follow the steps maybe the version is the issues 😉  Update, i checked and saw this,  Splunk Enterprise 9.4 and higher no longer support KV store server version 4.2. Upgrade to KV store server version 8.0 for continued support and security, and to comply with Splunk Support Policy. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/MigrateKVstore in the Admin manual to plan your upgrade. ... View more

Thanks for you reply, 1st solution is not my case. Trying to see the 2nd one but in my case i can login into the web page. ... View more