Splunk App for Windows Infrastructure: it has error "Key value store must be enabled. Please enable it"

[suwakhon]

Why am I getting error "Key value store must be enabled. Please enable it"
i use waindows server 2012 64 bit and Splunk version 6.2.3
I can't find a way to enable it. Please help, Thank you.

[mtime24]

I was experiencing this problem as well and it ended up being a permissions issue on the mongo folder itself. after reading through these posts as well as the referenced posts I did the following:

1) stopped splunk process
2) right clicked on mongo folder located in C:\Program Files\Splunk\var\lib\splunk\kvstore
3) selected security and altered the security permissions for local admin as well system, before exiting I selected the replace child object permissions entries with inheritable permissions entries from this object.
4) restarted splunk and my errors went away.

these steps cleared the error message Key value store must be enabled. Please enable it" and it enabled me to perform a KVStore migrate which i was not able to do before fixing this error. (KV Store initialization has failed.)

[malmoore]

Hi,

I know you said that you found the similar issue on a *nix system to be a dead end in your case, but this message appears only when the app detects that the App Key Value Store process (mongod) has not started for some reason.

Can you check to see if the mongod process is running on your system? Also, if you could check mongod.log and post the contents of the log during startup, that would be very helpful in determining next steps.

[RichING]

Mongod.exe is not running on the machine nor is any service of a similar name. The log on last restart is below:

2015-08-11T23:36:36.602Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-08-11T23:36:36.805Z [initandlisten] MongoDB starting : pid=5756 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1
2015-08-11T23:36:36.805Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-08-11T23:36:36.805Z [initandlisten] db version v2.6.7-splunk
2015-08-11T23:36:36.805Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550
2015-08-11T23:36:36.805Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015
2015-08-11T23:36:36.805Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-08-11T23:36:36.805Z [initandlisten] allocator: system
2015-08-11T23:36:36.805Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } }

---

old lock file: D:\Splunk\var\lib\splunk/kvstore\mongo\mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 23 dbs.
see: http://dochub.mongodb.org/core/repair for more information

---

2015-08-11T23:36:36.805Z [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2015-08-11T23:36:36.805Z [initandlisten] dbexit:
2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to close listening sockets...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to flush diaglog...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: going to close sockets...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: waiting for fs preallocator...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: lock for final commit...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: final commit...
2015-08-11T23:36:36.805Z [initandlisten] shutdown: closing all files...
2015-08-11T23:36:36.805Z [initandlisten] closeAllFiles() finished
2015-08-11T23:36:36.805Z [initandlisten] dbexit: really exiting now

It seems that there is a lock file which is similar to what was mentioned in this article.
http://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html

However, This is a production instance of splunk and OP did say that he did not recommend his fix to be used in production. Also I was unable to locate the specified lock file anywhere in the C:\Program Files\Splunk folder. Please let me know the best way to proceed.

Thank You,
Rich

[malmoore]

It's not going to be in C:\Program Files\Splunk unless that is where %SPLUNK_HOME% is. Like that other answer says, mongod (the Key Value Store service) won't run unless the lock file is zero bytes or not there because it thinks that the service is not in a good state. I would shut down Splunk on this instance, back up %SPLUNK_HOME%\var\lib\splunk, then delete the lock file and restart Splunk.

[RichING]

I was able to locate the mongod.lock file on a different drive. I performed the steps mentioned and removed the lock file (1KB in size). On splunk restart a new one was created that is 0kb. However I am still receiving the "Key value store must be enabled. Please enable it" in splunk. I re-traced my steps and will paste the mongod.log file below. It appears to me that there is a potential permissions issue with the file. Our splunkd service runs as a local system account. I found some linux articles when searching for the specific errors but nothing that seemed relevant. New log pasted below broken up by steps I had taken.

2015-08-13T17:40:17.665Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-08-13T17:40:17.949Z [initandlisten] MongoDB starting : pid=1732 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1
2015-08-13T17:40:17.949Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-08-13T17:40:17.949Z [initandlisten] db version v2.6.7-splunk
2015-08-13T17:40:17.949Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550
2015-08-13T17:40:17.949Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015
2015-08-13T17:40:17.949Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-08-13T17:40:17.949Z [initandlisten] allocator: system
2015-08-13T17:40:17.949Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } }
2015-08-13T17:40:17.996Z [initandlisten] journal dir=D:\Splunk\var\lib\splunk/kvstore\mongo\journal
2015-08-13T17:40:17.996Z [initandlisten] recover : no journal files present, no recovery needed
2015-08-13T17:40:18.437Z [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file D:\Splunk\var\lib\splunk/kvstore\mongo\journal\tempLatencyTest for writing errno:5 Access is denied.; returning false
2015-08-13T17:40:19.004Z [FileAllocator] allocating new datafile D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0, filling with zeroes...
2015-08-13T17:40:19.004Z [FileAllocator] creating directory D:\Splunk\var\lib\splunk/kvstore\mongo_tmp
2015-08-13T17:40:19.004Z [FileAllocator] FileAllocator: couldn't create D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 (D:\Splunk\var\lib\splunk/kvstore\mongo_tmp\1439487618012241) errno:5 Access is denied.
2015-08-13T17:40:19.004Z [FileAllocator] error: failed to allocate new file: D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 size: 16777216 . will try again in 10 seconds
2015-08-13T17:40:29.090Z [initandlisten] Assertion: 12520:new file allocation failure
2015-08-13T17:40:29.201Z [FileAllocator] allocating new datafile D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0, filling with zeroes...
2015-08-13T17:40:29.201Z [FileAllocator] FileAllocator: couldn't create D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 (D:\Splunk\var\lib\splunk/kvstore\mongo_tmp\1439487618012242) errno:5 Access is denied.
2015-08-13T17:40:29.201Z [FileAllocator] error: failed to allocate new file: D:\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tSessiV6ysHGNENqOVEZL92qEHLjZQ.0 size: 16777216 . will try again in 10 seconds
2015-08-13T17:40:29.295Z [initandlisten] exception in initAndListen: 12520 new file allocation failure, terminating
2015-08-13T17:40:29.295Z [initandlisten] dbexit:
2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to close listening sockets...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to flush diaglog...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: going to close sockets...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: waiting for fs preallocator...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: lock for final commit...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: final commit...
2015-08-13T17:40:29.295Z [initandlisten] shutdown: closing all files...
2015-08-13T17:40:29.295Z [initandlisten] closeAllFiles() finished
2015-08-13T17:40:29.295Z [initandlisten] journalCleanup...
2015-08-13T17:40:29.295Z [initandlisten] removeJournalFiles
2015-08-13T17:40:29.311Z [initandlisten] shutdown: removing fs lock...
2015-08-13T17:40:29.311Z [initandlisten] dbexit: really exiting now

---- After another restart of splunkd -----

2015-08-13T17:50:00.728Z warning: No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2015-08-13T17:50:00.932Z [initandlisten] MongoDB starting : pid=6604 port=8191 dbpath=D:\Splunk\var\lib\splunk/kvstore\mongo 64-bit host=SAC-CORP-SPLKH1
2015-08-13T17:50:00.932Z [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-08-13T17:50:00.932Z [initandlisten] db version v2.6.7-splunk
2015-08-13T17:50:00.932Z [initandlisten] git version: 7e66fa196686092ee1c184bd3f8fa1fe640c6550
2015-08-13T17:50:00.932Z [initandlisten] OpenSSL version: OpenSSL 1.0.1m-fips 19 Mar 2015
2015-08-13T17:50:00.932Z [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-08-13T17:50:00.932Z [initandlisten] allocator: system
2015-08-13T17:50:00.932Z [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "D:\Splunk\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "D:\Splunk\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } }
2015-08-13T17:50:00.963Z [initandlisten] exception in initAndListen: 13627 Unable to create/open lock file: D:\Splunk\var\lib\splunk/kvstore\mongo\mongod.lock Access is denied.. Is a mongod instance already running?, terminating
2015-08-13T17:50:00.963Z [initandlisten] dbexit:
2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to close listening sockets...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to flush diaglog...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: going to close sockets...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: waiting for fs preallocator...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: lock for final commit...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: final commit...
2015-08-13T17:50:00.963Z [initandlisten] shutdown: closing all files...
2015-08-13T17:50:00.963Z [initandlisten] closeAllFiles() finished
2015-08-13T17:50:00.963Z [initandlisten] dbexit: really exiting now

[malmoore]

Hi Rich,

Apologies for the late response. It's been really busy here and I didn't see the updates you had made until now.

So, I would see what is going on with the D:\ drive. Is it local? Is it a Windows drive formatted as NTFS, or a shared drive off a Linux server? If a Windows drive, then does the "Everyone" group have read permissions? Does the NT AUTHORITY\SYSTEM account have Full Control permissions? Once you fix the permissions issue, the service should start and this message should go away.

[RichING]

Experiencing the same problem. Splunk_x64-6.2.4 running on server 2012 with enterprise license. I have read the following articles but all are dead ends so far.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/AboutKVStore
http://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html
http://answers.splunk.com/answers/205689/splunk-app-for-windows-infrastructure-how-to-make.html
http://answers.splunk.com/answers/203979/splunk-app-for-microsoft-exchange-how-to-get-the-t.html

[RichING]

After working with Splunk Support I was able to resolve this issue. They were not able to reproduce it in the lab. It ended up being an odd permissions issue. I had to stop the splunkd service, go to our Splunk Data directory, right-click -> Properties ->Security -> Advanced, Click on the SYSTEM account -> Change permissions -> Check "Replace all child object permission entries" box. Basically this forced the SYSTEM account to reapply its permissions to all files. I then deleted the Mongod.lock file and _tmp directory in %Splunk_DATA%\var\lib\splunk\kvstore\mongo. Start splunk service

I am not sure why it happened this way but it is something to try if you are having a similar problem. We have a Splunk 6.0.3 instance that was upgraded to 6.2.4 running on Windows server 2012.

[alacercogitatus]

What is your license type? Enterprise, Trial, or Free?

Splunk Free does not allow KV Store, and may be the source of your problem. Otherwise, it is enabled by default.

[suwakhon]

My license type is Enterprise.

[MuS]

take a look at this answer http://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html