Splunk Enterprise, version 6.1.1, running on Win 2008 OS.
After running into weird permissions issues (no rights to edit files until we reset ownerships, "everyone" appearing on the filesystem), we decided to reset all ACL from the "root" Splunk folder (Drive:/Program Files/Splunk).
We disabled inheritance from Drive:/Program Files/Splunk and below.
Basically, it had the local administrators group, SYSTEM, and the splunk domain service account credentials with appropriate permissions on the whole Splunk file tree.
We dumped all ACL using dumpsec, to make sure it was all clean (it was).
Then, we started splunkd (running under a domain service account).
As soon as splunkd writes to a file, permissions are modified on the folder holding the file, and the file itself. Inheritance is broken on the folder containing the file written to, "everyone" is added, SYSTEM is removed, and any group different from local administrators is removed.
This is very annoying for the reasons listed below (and probably more which have not come up yet). The longer splunkd runs, the more folders and files get their permissions changed. After a week, it's an endless list of modified folders (all etc/apps/AppName and pretty much everything under it, conf files, lookups, views, etc)
- we do not want "everyone" to be set, on any folder or file, there is absolutely no reason for such permissions. It's all a Windows integrated security context, so "everyone" should not appear on any domain resource ACL.
- it generated loads of events reporting file permission changes
- it prevents appropriate auditing of the filesystem because if generates lots of noise
- it removes legitimate groups from accessing/editing files (such as lookups)
Has anyone met this type of behavior ?
We've seen such issues in older Splunk versions (reported bug in 4.x under windows), but they all date back awhile.
This is still an open issue as of Splunk 6.5.0 http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/KnownIssues
As a workaround you can run the command below to restore the inheritance. It leaves the explicit permission assignments that splunkd does as part of it's normal operation so it should be low risk. It can take a while to run if you include the whole Splunk directory so adapt to suit your needs.
icacls.exe "C:\Program Files\Splunk\etc" /inheritance:e /T
You can use the /reset option of the icacls to restore the inheritance and remove the explicit permission assignments but I haven't tested the impact of that.
I make it a habit to reset perms before every code upgrade now using something similar
icacls "D:\Splunk\*" /reset /Q /C /T
I've also noticed the recent 6.3 and 6.4.x Enterprise MSI installers also run an icalcs command which grants the builtin\admins full control over a few directories. here is one of them in action:
I have observed the same behavior running 6.2 on Win2008. This makes it difficult to provision a server so a non-administrator can access and modify Splunk configuration files. The desired behavior would be that if a user account or user group is given explicit permissions to a "splunk" folder or a "splunk" file that these permissions would persist after "splunk" modifies the file or folder.
The ACL override appears to be a bug. Is it?