Unable to view Splunk KV store status

[Sathish28]

We are migrating the Splunk 9.0.3 Search Head from Virtual box to Physical box.
Splunk services were up and running in new Physical box but in Splunk Web UI, I was unable to login using the
my authorized credentials and found the below error in Splunkd.log
 
01-21-2025 05:18:05.218 -0500 ERROR ExecProcessor [3275615 ExecProcessor] - message from "/apps/splunk/splunk/etc/apps/splunk_app_db_connect/bin/server.sh" action=task_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

[isoutamo]

Hi

as @VatsalJagani already said that error message didn’t relate to you login issue. It’s just told that your DB connect didn’t work as kvstore is somehow broken/stop.

On splunkd.log should be some lines which could help us to see what was a real issue.

But let’s start that migration part as it’s quite obvious that it has something to do with this issue!

From where you migrated it and what is target environment?

How do you do the migration?

Was there any issues before migration?

Anything else we should know?

r. Ismo

[kiran_panchavat]

@Sathish28 

1. Check status of KV store

2. Verify the status of the KV Store service

./splunk show kvstore-status

3. Check mongod.log

less /opt/splunk/var/log/splunk/mongod.log

4. Verify that the permissions for the KV Store directories and files are set correctly. Incorrect permissions can prevent the KV Store from initializing.

• Set splunk.key to the default file permission.
  chmod 600 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
  Restart Splunk

[VatsalJagani]

@Sathish28- Few things I want to take your attention:

• The error you are seeing is not related to the login issue you are having at all.

For the Login Issue:

1. Are you trying LDAP credential?
   1. Login first with Admin Splunk native account.
   2. Then fix the LDAP related issue. Check Splunk internal logs & LDAP configuration page.
2. Is it Splunk native authentication?
   1. Then you might need to reset the creds.

For Mongod related errors you are seeing in the logs. As suggested by @splunkreal  please check the Splunk's internal logs to find the details on why mongodb service unable to start.

I hope this helps!!! Kindly upvote if it does!!!

[splunkreal]

check mongod.log under $SPLUNK_HOME/var/log/splunk/