Knowledge Management

Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

madhufuture
New Member

I'm running into this issue consistently when ever I change the logon details of "Splunkd Service" to a domain account. When the service is running on Local System account, "Splunk DB Connect" is fine.

I'm on Windows machine.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Hi @madhufuture,
Try starting Splunk with Administrator permission. If that does not work then try removing below file,

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.lock

Restart the Splunk, now hopefully kvstore will start.

0 Karma

madhufuture
New Member

I'm not getting the error messages after deleting the mongod.lock and restarting the splunk service. But now I'm getting "DBX Server is not available, please make sure it is started and listening on 9998 " error when I'm trying to configure a new connection in Splunk DB Connect.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Looks like firewall issue, check for port 9998 should be open from both the side, Splunk and DBX side. Port 9998 should be publicly open. Try this!
Make sure you have configured Splunk DB Connect correctly - http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/ConfigureDBConnectsettings

0 Karma

madhufuture
New Member

Ok, What do you mean by DBX side? I have installed both Splunk Enterprise and Splunk DB Connect on the same server. And I'm trying to connect to a remote SQL Server. I have opened port 9998 on the
Splunk server.
Then when I'm trying to create a new connection, I am getting this error.

0 Karma

deepashri_123
Motivator

Hey@madhufuture,

Please search the internal logs for exact errors that can help you troubleshoot.
Run the following query:
index=_internal log_level=ERROR

Let me know if this helps!!

0 Karma

madhufuture
New Member

Hi @deepashri_123 Thanks for your response.
I have checked the logs based on the search query which you provided. I see the following error

KVStoreConfigurationProvider - Could not get ping from mongod.
KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

Could you please let me know how I can fix this issue.

0 Karma

deepashri_123
Motivator

Hey@madhufuture,

Could you check this answer, seems like certificate issue:
https://answers.splunk.com/answers/314499/after-upgrading-to-splunk-630-why-am-i-getting-the.html
Let me know if this helps!!

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...