Deployment Architecture

KVStore does not start when running Splunk 9.4

MaverickT
Communicator

I am posting this to maybe save you from few hours of troubleshooting like I did.

I did clean install of Splunk 9.4 in small customer environment with virtualized AIO instance. After the installation there was an error notifying that KV Store can not start and that mongo log should be checked.

The following error was logged:

 

ERROR KVStoreConfigurationProvider [4755 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.

 

 
Mongod.log was completely empty.  So there was no clues in the log files about what is wrong and what can I do to make KVStore operational.

Time to start Googling. Solution will be posted in the next post.

0 Karma
1 Solution

MaverickT
Communicator

I did some reading of the documentation and realized that underlying Mongo DB was upgraded to 7. I figured out that Mongo DB 5+ requires AVX instruction set

So time to check if CPU supports AVX instruction set - in my case the CPU model did support this instructions. But running the lscpu command didnt show AVX flags. It turned out that AVX instructions were not available, because the VM had Processor compatibility mode enabled. In hyper-v we had to remove "Allow migration to a virtual machine host with a different processor version" checkbox.  After VM was restarted, AVX appeared in CPU flags and Splunk KV Store was operational.

Screenshot 2025-01-08 at 22.23.30.png

Lession learned: before upgrading  to 9.4 (or making fresh install), check if AVX flag is available. If it isn't, it is about time to upgrade your hardware 😁 and in stick to Splunk 9.3.

View solution in original post

isoutamo
SplunkTrust
SplunkTrust

Have you checked from mongodb.log why this is not starting? There is one another case where Windows OS was not supported by Splunk 9.4.0 version. https://community.splunk.com/t5/Splunk-Enterprise/KVstore-unable-to-start-after-upgrade-to-Splunk-En...

0 Karma

MaverickT
Communicator

I did some reading of the documentation and realized that underlying Mongo DB was upgraded to 7. I figured out that Mongo DB 5+ requires AVX instruction set

So time to check if CPU supports AVX instruction set - in my case the CPU model did support this instructions. But running the lscpu command didnt show AVX flags. It turned out that AVX instructions were not available, because the VM had Processor compatibility mode enabled. In hyper-v we had to remove "Allow migration to a virtual machine host with a different processor version" checkbox.  After VM was restarted, AVX appeared in CPU flags and Splunk KV Store was operational.

Screenshot 2025-01-08 at 22.23.30.png

Lession learned: before upgrading  to 9.4 (or making fresh install), check if AVX flag is available. If it isn't, it is about time to upgrade your hardware 😁 and in stick to Splunk 9.3.

mwk1000
Path Finder

I also hit an upgrade bug with 9.4.1 on a clients indexers , the upgrade migration mongo 4-7 failed to run due to the scripts not using SPLUNK_DB but hardcoding /opt/splunk/var/lib.... The indexers had a separate filesystem /data01/,,,, I was able to create a link from the mongo under /opt/splunk/var/lib/splunk/kvstore... to the "real" one in /data01 and restart triggering the upgrade process to complete properly ....

0 Karma

mwk1000
Path Finder

So if your indexers have separate storage filesystem for indexes consider pre upgrade creating the links ln -s /mypath-to/mongo /opt/splunk/var/lib/splunk/kvstore/mongo for a headach free kvstore update

 

 

Get Updates on the Splunk Community!

Index This | What did the zero say to the eight?

June 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

Splunk Observability Cloud's AI Assistant in Action Series: Onboarding New Hires & ...

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Now Playing: Splunk Education Summer Learning Premieres

It’s premiere season, and Splunk Education is rolling out new releases you won’t want to miss. Whether you’re ...