Subject: KV Store fails after upgrade to Splunk 10.2.2 – Unsupported WiredTiger file version Hi Team, We recently upgraded our environment from Splunk Enterprise 9.3.5 to 10.2.2. After the upgrade, we observed intermittent issues where services appear as down in the UI, even though the backend services are running normally. During troubleshooting, we found the following errors in mongod.log: Detected unclean shutdown - /opt/data/kvstore/mongo/mongod.lock is not empty.
Recovering data from the last clean checkpoint.
WiredTiger error (-31802):
unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5
WT_ERROR: non-specific WiredTiger error Additional details: Previous Splunk version: 9.3.5 Current Splunk version: 10.2.2 KV Store database path: /opt/data/kvstore/mongo The upgrade completed successfully, and KV Store was upgraded as part of the process. The issue was first noticed after the upgrade. We are also seeing intermittent health/status reporting issues in the UI. Could anyone advise on: What typically causes the "unsupported WiredTiger file version" error after a Splunk upgrade? Does this indicate that an older MongoDB/KV Store binary is being used against upgraded KV Store files? Are there any known issues with custom KV Store locations (/opt/data/kvstore/mongo) during upgrades? What are the recommended validation and recovery steps to confirm KV Store integrity and compatibility? Any guidance or similar experiences would be greatly appreciated. Thank you.
... View more