topic KVStore does not start when running Splunk 9.4 in Deployment Architecture

KVStore does not start when running Splunk 9.4

MaverickT — Wed, 08 Jan 2025 21:45:03 GMT

I am posting this to maybe save you from few hours of troubleshooting like I did.

I did clean install of Splunk 9.4 in small customer environment with virtualized AIO instance. After the installation there was an error notifying that KV Store can not start and that mongo log should be checked.

The following error was logged:

ERROR KVStoreConfigurationProvider [4755 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.

Mongod.log was completely empty. So there was no clues in the log files about what is wrong and what can I do to make KVStore operational.

Time to start Googling. Solution will be posted in the next post.

Re: KVStore does not start when running Splunk 9.4

MaverickT — Wed, 08 Jan 2025 21:54:39 GMT

I did some reading of the documentation and realized that underlying Mongo DB was upgraded to 7. I figured out that Mongo DB 5+ requires AVX instruction set.

So time to check if CPU supports AVX instruction set - in my case the CPU model did support this instructions. But running the lscpu command didnt show AVX flags. It turned out that AVX instructions were not available, because the VM had Processor compatibility mode enabled. In hyper-v we had to remove "Allow migration to a virtual machine host with a different processor version" checkbox. After VM was restarted, AVX appeared in CPU flags and Splunk KV Store was operational.

Lession learned: before upgrading to 9.4 (or making fresh install), check if AVX flag is available. If it isn't, it is about time to upgrade your hardware 😁 and in stick to Splunk 9.3.

Re: KVStore does not start when running Splunk 9.4

isoutamo — Wed, 08 Jan 2025 21:56:21 GMT

Have you checked from mongodb.log why this is not starting? There is one another case where Windows OS was not supported by Splunk 9.4.0 version. https://community.splunk.com/t5/Splunk-Enterprise/KVstore-unable-to-start-after-upgrade-to-Splunk-Enterprise-9-4/m-p/708264#M21264

Re: KVStore does not start when running Splunk 9.4

mwk1000 — Mon, 12 May 2025 18:31:10 GMT

I also hit an upgrade bug with 9.4.1 on a clients indexers , the upgrade migration mongo 4-7 failed to run due to the scripts not using SPLUNK_DB but hardcoding /opt/splunk/var/lib.... The indexers had a separate filesystem /data01/,,,, I was able to create a link from the mongo under /opt/splunk/var/lib/splunk/kvstore... to the "real" one in /data01 and restart triggering the upgrade process to complete properly ....

Re: KVStore does not start when running Splunk 9.4

mwk1000 — Mon, 12 May 2025 18:39:03 GMT

So if your indexers have separate storage filesystem for indexes consider pre upgrade creating the links ln -s /mypath-to/mongo /opt/splunk/var/lib/splunk/kvstore/mongo for a headach free kvstore update

Re: KVStore does not start when running Splunk 9.4

dpollardcouk — Fri, 04 Jul 2025 12:23:32 GMT

Which if your indexers are using a different partition for their storage, could be anywhere. I found that I was missing the link too, put note that I've put the link in at kvstore level rather than the mongo

ln -s /splunkdata/kvstore /opt/splunk/var/lib/splunk/kvstore

Where /splunkdata/ is my mounted data drive where all my indexes go.

Re: KVStore does not start when running Splunk 9.4

kundeng — Mon, 28 Jul 2025 00:31:39 GMT

This breaks Splunk running in Rosetta on ARM based M1,M2,M3,M4 mac computers.

Previously Splunk can be run smoothly in Rosetta emulated Linux VM on new macs.

Re: KVStore does not start when running Splunk 9.4

trashyroadz — Mon, 22 Sep 2025 17:40:21 GMT

I realize this response comes in a bit late in the game for most on this thread, but hopefully it will be useful to someone.

KV Store is not used on indexers. In fact, it can be disabled on any Splunk Enterprise instance that is not running as a search head (see caveats below). This means includes Indexers, Deployment Servers, Cluster Managers, License Managers and Search Head Cluster Deployers, Heavy Forwarders (see exceptions below).

Instances that SHOULD run KV Store include:

Search Heads
Monitoring Console (if you want access to its KV Store dashboards-- see caveats below)
Heavy Forwarders, if used as search heads or are running DB Connect app (or, not likely but possible, any custom app that specifically relies on the KV Store)
Lastly, any Splunk Enterprise instance in your deployment which requires token-based authentication

CAVEATS:

If your Monitoring Console machine also operates another Splunk role such as Deployment Server or License Manager, keep the KV Store running on that instance.
As of this writing, the Monitoring Console appears to use the KV Store only for KV Store-specific dashboards. If you don't care about those dashboards and the KV Store is giving you real headaches on that instance, disabling it will cause the dashboards which rely on KV Store to stop working, but all other Monitoring Console dashboards will function as expected.
Universal Forwarders are not shipped with the KV Store so nothing to worry about there