Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

[madhufuture]

I'm running into this issue consistently when ever I change the logon details of "Splunkd Service" to a domain account. When the service is running on Local System account, "Splunk DB Connect" is fine.

I'm on Windows machine.

[VatsalJagani]

Hi @madhufuture,
Try starting Splunk with Administrator permission. If that does not work then try removing below file,

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.lock

Restart the Splunk, now hopefully kvstore will start.

[madhufuture]

I'm not getting the error messages after deleting the mongod.lock and restarting the splunk service. But now I'm getting "DBX Server is not available, please make sure it is started and listening on 9998 " error when I'm trying to configure a new connection in Splunk DB Connect.

[VatsalJagani]

Looks like firewall issue, check for port 9998 should be open from both the side, Splunk and DBX side. Port 9998 should be publicly open. Try this!
Make sure you have configured Splunk DB Connect correctly - http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/ConfigureDBConnectsettings

[madhufuture]

Ok, What do you mean by DBX side? I have installed both Splunk Enterprise and Splunk DB Connect on the same server. And I'm trying to connect to a remote SQL Server. I have opened port 9998 on the
Splunk server.
Then when I'm trying to create a new connection, I am getting this error.

[deepashri_123]

Hey@madhufuture,

Please search the internal logs for exact errors that can help you troubleshoot.
Run the following query:
index=_internal log_level=ERROR

Let me know if this helps!!

[madhufuture]

Hi @deepashri_123 Thanks for your response.
I have checked the logs based on the search query which you provided. I see the following error

KVStoreConfigurationProvider - Could not get ping from mongod.
KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

Could you please let me know how I can fix this issue.

[deepashri_123]

Hey@madhufuture,

Could you check this answer, seems like certificate issue:
https://answers.splunk.com/answers/314499/after-upgrading-to-splunk-630-why-am-i-getting-the.html
Let me know if this helps!!