Our general policy is to not run applications on our servers as the "root" user. However, some log files get written with the read/write access to only the owner (not to any group, and not to others), and the owner is always "root".
When attempting to add a monitor stanza on this file in the inputs.conf for the forwarder, I get the following error:
Insufficient permissions to read file='/file/location/filename' (hint: Permission denied).
(The file in question is an SMTP maillog.)
Permissions on the file:
-rw------- root root
As I understand it, one of the perks/features of Splunk is that the non-root user (in this case, "splunk") should be able to be assigned read permissions to all files, even ones that are owned by root.
Changing the permissions on the file won't work because each night when the log rolls over, the permissions will get reset.
What's the way to get around this? If I remember correctly from things I've heard (and a few of the other Answers posts allude to this), there's some file on the host that can be modified to add the "splunk" user and grant read-only access to these root-owned files, but I don't know/understand the specifics of how to do that.
The forwarder is running on a Linux CentOS 5.11 system. The "splunk" user is a local user.
You can give user
splunk sudo by adding this entry AT THE BOTTOM of this file
splunk ALL = (root) NOPASSWD: /opt/splunk/bin/splunk
Then run Splunk like this:
sudo /opt/splunk/bin/splunk start
The other thing you can do is try to make sure that new files get
+gr ("Group Read) permission in that directory (assuming user
splunk is in the
root group). The group ownership can be inherited by new files and folders created in your folder /path/to/parent by setting the setgid bit using
chmod g+s like this:
chmod g+s /file/location/
Now, all new files and folder created under
/file/location/ will have the same group assigned as is set on
POSIX file permissions are not inherited; they are given by the creating process and combined with its current
umask value and you can use POSIX ACLs to control this; to set the default ACL on a directory:
setfacl -d -m u::rwX,g::rX,o::- /file/location/
This will apply
setfacl to the
/file/location/ directory, -modifying the -default ACLs – those that will be applied to newly created items. (Uppercase X means only directories will receive the +x bit.)
Also, If necessary, you can add a u:someuser:rwX or g:someuser:rwX – preferably a group – to the ACLs.
Yes, running Splunk as the root user is a problem.
We can't easily change the file permissions because it would require altering our application (we have all sorts of custom stuff under the hood)...and it's a piece of code that hasn't even been looked at by anyone in at least 5 years, probably longer. The current rotation resets permissions back to what I've listed in the post.
The last solution does not require you to modify the application; unless the application specifies/enforces the permissions (only one way to find out), this will work because it is modifying the OS, not the application. You have nothing to lose by trying.
If it's syslog, what daemon are you running? You should be able to modify the permissions of the output file via the syslog daemon, not your app.
Splunk cannot override Unix security. If it could, that would be a Very Bad Thing. For Splunk to read a log, it must have permission to do so.
Therefore, if your logs are root:root 600, then you have two options: Run Splunk as root, or change the permissions on the logs.
Running a Splunk forwarder as root is somewhat common because of these permissions quandaries. It's simply an agent running locally which does not need to accept any incoming connections. (Everything a forwarder does is outbound initiated) Basically, the exposure of running a forwarder as root is relatively minimal. (Compared to the Splunk server which I would never recommend running as root) Yes, if Splunk is compromised the host is effectively rooted. But you have to be local to compromise Splunk, meaning you're likely rooted anyway.
There is also the fact that your application is writing logs to root:root 600 in the first place, which means your app runs as root. Just as much of a security risk as running Splunk as root, to be honest. That may not matter, but it's a precedence you can point to.
One note to @emiller42's comment: Actually you would not necessarily have to be local to compromise a UF. Just like full blown Splunk it opens port 8089 for the REST API. Now there are options around firewalls, turning this off, and other protections that could be taken, but by itself, a UF running as root opens a service as root that assuming the existence of an exploit, could be compromised and root the box.