I just upgraded Splunk to 10.4 (10.2.2 x86_64 to 10.4 arm64) on macOS, and the previously working KVStore now does not start. Seems it's trying and failing to import a certificate into the system keychain, and then mongod does not get started. I tried using my own certificate and the default certificate, both have failed. Any ways to fix this? (I understand KVStore might be unsupported on macOS but it was running fine in a previous version, and I had a working Splunk mobile connection, which now no longer works.) 05-26-2026 21:52:24.168 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Starting mongod with executable name=mongod-8.0 version=kvstore version 8.0 05-26-2026 21:52:24.169 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Setting env var LC_ALL=C 05-26-2026 21:52:24.169 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --dbpath /Volumes/Server Data/Splunk/kvstore/mongo 05-26-2026 21:52:24.169 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --storageEngine wiredTiger 05-26-2026 21:52:24.169 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using cacheSize=3.5999999999999996GB 05-26-2026 21:52:24.169 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --port 8191 05-26-2026 21:52:24.170 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc 05-26-2026 21:52:24.170 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200 05-26-2026 21:52:24.170 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --keyFile /Volumes/Server Data/Splunk/kvstore/mongo/splunk.key 05-26-2026 21:52:24.171 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0 05-26-2026 21:52:24.171 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0 05-26-2026 21:52:24.171 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --setParameter minSnapshotHistoryWindowInSeconds=5 05-26-2026 21:52:24.171 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --setParameter ocspEnabled=true 05-26-2026 21:52:24.171 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --replSet 809AB6C9-2B98-484A-84A2-23B7BAF22D08 05-26-2026 21:52:24.172 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --ipv6 05-26-2026 21:52:24.172 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --bind_ip=::,0.0.0.0 (all ip addresses) 05-26-2026 21:52:24.172 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --sslCAFile /opt/splunk/etc/auth/cacert.pem 05-26-2026 21:52:24.172 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL 05-26-2026 21:52:24.172 +1000 INFO MongodRunner [30073074 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames 05-26-2026 21:52:24.198 +1000 ERROR MongodRunner [30073074 KVStoreConfigurationThread] - Failed to import PFX file into Keychain 05-26-2026 21:52:24.198 +1000 ERROR MongodRunner [30073074 KVStoreConfigurationThread] - Failed to add PFX to certificate store 05-26-2026 21:52:24.198 +1000 WARN MongoClient [30073074 KVStoreConfigurationThread] - Disabling TLS hostname validation for localhost 05-26-2026 21:52:24.437 +1000 WARN IntrospectionGenerator:resource_usage [30073152 ExecProcessor] - MongoClient - Disabling TLS hostname validation for localhost 05-26-2026 21:52:24.437 +1000 INFO WatchedFile [30073198 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/mongod.log'. 05-26-2026 21:52:24.455 +1000 INFO WatchedFile [30073198 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/mongod_upgrade.log'. 05-26-2026 21:57:26.226 +1000 ERROR KVStoreConfigurationProvider [30073074 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=302026ms 05-26-2026 21:57:26.226 +1000 ERROR KVStoreConfigurationProvider [30073074 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed. 05-26-2026 21:57:26.226 +1000 ERROR KVStoreBulletinBoardManager [30073074 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details.. 05-26-2026 21:57:26.226 +1000 ERROR KVStoreBulletinBoardManager [30073074 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details. 05-26-2026 21:57:26.226 +1000 INFO KVStoreConfigurationProvider [30073074 KVStoreConfigurationThread] - Mongod service shutting down   ... View more

Hi all,  I'm currently running arm64 version of Splunk on my M4 Mac mini, and I noticed KV store isn't available. Digging deeper it seems like mongo binaries are missing from the arm64 release but are present in the x86_64 release, and works just fine under Rosetta translation. Why is this and can I somehow add mongodb binaries back? I can see from https://www.mongodb.com/docs/manual/administration/production-notes/ that mongodb does support arm64 (armv8.2+). ... View more