After working with Splunk Support I was able to resolve this issue. They were not able to reproduce it in the lab. It ended up being an odd permissions issue. I had to stop the splunkd service, go to our Splunk Data directory, right-click -> Properties ->Security -> Advanced, Click on the SYSTEM account -> Change permissions -> Check "Replace all child object permission entries" box. Basically this forced the SYSTEM account to reapply its permissions to all files. I then deleted the Mongod.lock file and _tmp directory in %Splunk_DATA%\var\lib\splunk\kvstore\mongo. Start splunk service I am not sure why it happened this way but it is something to try if you are having a similar problem. We have a Splunk 6.0.3 instance that was upgraded to 6.2.4 running on Windows server 2012. ... View more

I was able to locate the mongod.lock file on a different drive. I performed the steps mentioned and removed the lock file (1KB in size). On splunk restart a new one was created that is 0kb. However I am still receiving the "Key value store must be enabled. Please enable it" in splunk. I re-traced my steps and will paste the mongod.log file below. It appears to me that there is a potential permissions issue with the file. Our splunkd service runs as a local system account. I found some linux articles when searching for the specific errors but nothing that seemed relevant. New log pasted below broken up by steps I had taken. 2015-08-13T17:40:17.665Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter 2015-08-13T17:40:17.949Z [initandlisten] MongoDB starting : pid=1732 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1 2015-08-13T17:40:17.949Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2 2015-08-13T17:40:17.949Z [initandlisten] db version v2.6.7-splunk 2015-08-13T17:40:17.949Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550 2015-08-13T17:40:17.949Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015 2015-08-13T17:40:17.949Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49 2015-08-13T17:40:17.949Z [initandlisten] allocator: system 2015-08-13T17:40:17.949Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } } 2015-08-13T17:40:17.996Z [initandlisten] journal dir=D:\Splunk\var\lib\splunk/kvstore\mongo\journal 2015-08-13T17:40:17.996Z [initandlisten] recover : no journal files present, no recovery needed 2015-08-13T17:40:18.437Z [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file D:\Splunk\var\lib\splunk/kvstore\mongo\journal\tempLatencyTest for writing errno:5 Access is denied.; returning false 2015-08-13T17:40:19.004Z [FileAllocator] allocating new datafile D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0, filling with zeroes... 2015-08-13T17:40:19.004Z [FileAllocator] creating directory D:\Splunk\var\lib\splunk/kvstore\mongo_tmp 2015-08-13T17:40:19.004Z [FileAllocator] FileAllocator: couldn't create D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 (D:\Splunk\var\lib\splunk/kvstore\mongo_tmp\1439487618012241) errno:5 Access is denied. 2015-08-13T17:40:19.004Z [FileAllocator] error: failed to allocate new file: D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 size: 16777216 . will try again in 10 seconds 2015-08-13T17:40:29.090Z [initandlisten] Assertion: 12520:new file allocation failure 2015-08-13T17:40:29.201Z [FileAllocator] allocating new datafile D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0, filling with zeroes... 2015-08-13T17:40:29.201Z [FileAllocator] FileAllocator: couldn't create D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 (D:\Splunk\var\lib\splunk/kvstore\mongo_tmp\1439487618012242) errno:5 Access is denied. 2015-08-13T17:40:29.201Z [FileAllocator] error: failed to allocate new file: D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 size: 16777216 . will try again in 10 seconds 2015-08-13T17:40:29.295Z [initandlisten] exception in initAndListen: 12520 new file allocation failure, terminating 2015-08-13T17:40:29.295Z [initandlisten] dbexit: 2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to close listening sockets... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to flush diaglog... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to close sockets... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: waiting for fs preallocator... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: lock for final commit... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: final commit... 2015-08-13T17:40:29.295Z [initandlisten] shutdown: closing all files... 2015-08-13T17:40:29.295Z [initandlisten] closeAllFiles() finished 2015-08-13T17:40:29.295Z [initandlisten] journalCleanup... 2015-08-13T17:40:29.295Z [initandlisten] removeJournalFiles 2015-08-13T17:40:29.311Z [initandlisten] shutdown: removing fs lock... 2015-08-13T17:40:29.311Z [initandlisten] dbexit: really exiting now ---- After another restart of splunkd ----- 2015-08-13T17:50:00.728Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter 2015-08-13T17:50:00.932Z [initandlisten] MongoDB starting : pid=6604 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1 2015-08-13T17:50:00.932Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2 2015-08-13T17:50:00.932Z [initandlisten] db version v2.6.7-splunk 2015-08-13T17:50:00.932Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550 2015-08-13T17:50:00.932Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015 2015-08-13T17:50:00.932Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49 2015-08-13T17:50:00.932Z [initandlisten] allocator: system 2015-08-13T17:50:00.932Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } } 2015-08-13T17:50:00.963Z [initandlisten] exception in initAndListen: 13627 Unable to create/open lock file: D:\Splunk\var\lib\splunk/kvstore\mongo\mongod.lock Access is denied.. Is a mongod instance already running?, terminating 2015-08-13T17:50:00.963Z [initandlisten] dbexit: 2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to close listening sockets... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to flush diaglog... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to close sockets... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: waiting for fs preallocator... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: lock for final commit... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: final commit... 2015-08-13T17:50:00.963Z [initandlisten] shutdown: closing all files... 2015-08-13T17:50:00.963Z [initandlisten] closeAllFiles() finished 2015-08-13T17:50:00.963Z [initandlisten] dbexit: really exiting now ... View more

Mongod.exe is not running on the machine nor is any service of a similar name. The log on last restart is below: 2015-08-11T23:36:36.602Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter 2015-08-11T23:36:36.805Z [initandlisten] MongoDB starting : pid=5756 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1 2015-08-11T23:36:36.805Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2 2015-08-11T23:36:36.805Z [initandlisten] db version v2.6.7-splunk 2015-08-11T23:36:36.805Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550 2015-08-11T23:36:36.805Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015 2015-08-11T23:36:36.805Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49 2015-08-11T23:36:36.805Z [initandlisten] allocator: system 2015-08-11T23:36:36.805Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } } old lock file: D:\Splunk\var\lib\splunk/kvstore\mongo\mongod.lock. probably means unclean shutdown, but there are no journal files to recover. this is likely human error or filesystem corruption. please make sure that your journal directory is mounted. found 23 dbs. see: http://dochub.mongodb.org/core/repair for more information 2015-08-11T23:36:36.805Z [initandlisten] exception in initAndListen: 12596 old lock file, terminating 2015-08-11T23:36:36.805Z [initandlisten] dbexit: 2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to close listening sockets... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to flush diaglog... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to close sockets... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: waiting for fs preallocator... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: lock for final commit... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: final commit... 2015-08-11T23:36:36.805Z [initandlisten] shutdown: closing all files... 2015-08-11T23:36:36.805Z [initandlisten] closeAllFiles() finished 2015-08-11T23:36:36.805Z [initandlisten] dbexit: really exiting now It seems that there is a lock file which is similar to what was mentioned in this article. http://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html However, This is a production instance of splunk and OP did say that he did not recommend his fix to be used in production. Also I was unable to locate the specified lock file anywhere in the C:\Program Files\Splunk folder. Please let me know the best way to proceed. Thank You, Rich ... View more

Experiencing the same problem. Splunk_x64-6.2.4 running on server 2012 with enterprise license. I have read the following articles but all are dead ends so far. http://docs.splunk.com/Documentation/Splunk/latest/Admin/AboutKVStore http://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html http://answers.splunk.com/answers/205689/splunk-app-for-windows-infrastructure-how-to-make.html http://answers.splunk.com/answers/203979/splunk-app-for-microsoft-exchange-how-to-get-the-t.html ... View more