topic Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine in Knowledge Management

Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

madhufuture — Thu, 27 Sep 2018 00:28:28 GMT

I'm running into this issue consistently when ever I change the logon details of "Splunkd Service" to a domain account. When the service is running on Local System account, "Splunk DB Connect" is fine.

I'm on Windows machine.

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

deepashri_123 — Tue, 29 Sep 2020 21:23:49 GMT

Hey@madhufuture,

Please search the internal logs for exact errors that can help you troubleshoot.
Run the following query:
index=_internal log_level=ERROR

Let me know if this helps!!

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

madhufuture — Thu, 27 Sep 2018 12:42:41 GMT

Hi @deepashri_123 Thanks for your response.
I have checked the logs based on the search query which you provided. I see the following error

KVStoreConfigurationProvider - Could not get ping from mongod.
KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.

Could you please let me know how I can fix this issue.

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

VatsalJagani — Thu, 27 Sep 2018 14:22:38 GMT

Hi @madhufuture,
Try starting Splunk with Administrator permission. If that does not work then try removing below file,

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.lock

Restart the Splunk, now hopefully kvstore will start.

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

madhufuture — Thu, 27 Sep 2018 15:52:07 GMT

I'm not getting the error messages after deleting the mongod.lock and restarting the splunk service. But now I'm getting "DBX Server is not available, please make sure it is started and listening on 9998 " error when I'm trying to configure a new connection in Splunk DB Connect.

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

VatsalJagani — Fri, 28 Sep 2018 04:08:38 GMT

Looks like firewall issue, check for port 9998 should be open from both the side, Splunk and DBX side. Port 9998 should be publicly open. Try this!
Make sure you have configured Splunk DB Connect correctly - http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/ConfigureDBConnectsettings

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

deepashri_123 — Fri, 28 Sep 2018 04:38:58 GMT

Hey@madhufuture,

Could you check this answer, seems like certificate issue:
https://answers.splunk.com/answers/314499/after-upgrading-to-splunk-630-why-am-i-getting-the.html
Let me know if this helps!!

Re: Failed to start KV Store process. See mongod.log and splunkd.log for details - Windows machine

madhufuture — Fri, 28 Sep 2018 12:30:51 GMT

Ok, What do you mean by DBX side? I have installed both Splunk Enterprise and Splunk DB Connect on the same server. And I'm trying to connect to a remote SQL Server. I have opened port 9998 on the
Splunk server.
Then when I'm trying to create a new connection, I am getting this error.