Security
Highlighted

How to get "splunk" user to read "root" user-owned files?

Builder

Our general policy is to not run applications on our servers as the "root" user. However, some log files get written with the read/write access to only the owner (not to any group, and not to others), and the owner is always "root".

When attempting to add a monitor stanza on this file in the inputs.conf for the forwarder, I get the following error:

Insufficient permissions to read file='/file/location/filename' (hint: Permission denied).

(The file in question is an SMTP maillog.)

Permissions on the file:

-rw------- root root

As I understand it, one of the perks/features of Splunk is that the non-root user (in this case, "splunk") should be able to be assigned read permissions to all files, even ones that are owned by root.

Changing the permissions on the file won't work because each night when the log rolls over, the permissions will get reset.

What's the way to get around this? If I remember correctly from things I've heard (and a few of the other Answers posts allude to this), there's some file on the host that can be modified to add the "splunk" user and grant read-only access to these root-owned files, but I don't know/understand the specifics of how to do that.

The forwarder is running on a Linux CentOS 5.11 system. The "splunk" user is a local user.

Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Esteemed Legend

You can give user splunk sudo by adding this entry AT THE BOTTOM of this file /etc/sudoers:

splunk ALL = (root) NOPASSWD: /opt/splunk/bin/splunk

Then run Splunk like this:

sudo /opt/splunk/bin/splunk start
0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Builder

Won't that run Splunk as the "root" user, though, not as the "splunk" user?

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Esteemed Legend

The other thing you can do is try to make sure that new files get +gr ("Group Read) permission in that directory (assuming user splunk is in the root group). The group ownership can be inherited by new files and folders created in your folder /path/to/parent by setting the setgid bit using chmod g+s like this:

chmod g+s /file/location/

Now, all new files and folder created under /file/location/ will have the same group assigned as is set on /file/location

POSIX file permissions are not inherited; they are given by the creating process and combined with its current umask value and you can use POSIX ACLs to control this; to set the default ACL on a directory:

setfacl -d -m u::rwX,g::rX,o::- /file/location/

This will apply setfacl to the /file/location/ directory, -modifying the -default ACLs – those that will be applied to newly created items. (Uppercase X means only directories will receive the +x bit.)

Also, If necessary, you can add a u:someuser:rwX or g:someuser:rwX – preferably a group – to the ACLs.

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Builder

Yes, running Splunk as the root user is a problem.

We can't easily change the file permissions because it would require altering our application (we have all sorts of custom stuff under the hood)...and it's a piece of code that hasn't even been looked at by anyone in at least 5 years, probably longer. The current rotation resets permissions back to what I've listed in the post.

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Esteemed Legend

The last solution does not require you to modify the application; unless the application specifies/enforces the permissions (only one way to find out), this will work because it is modifying the OS, not the application. You have nothing to lose by trying.

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Champion

If it's syslog, what daemon are you running? You should be able to modify the permissions of the output file via the syslog daemon, not your app.

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Builder

This log is being written by custom code in our application, not syslog.

0 Karma
Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Motivator

Splunk cannot override Unix security. If it could, that would be a Very Bad Thing. For Splunk to read a log, it must have permission to do so.

Therefore, if your logs are root:root 600, then you have two options: Run Splunk as root, or change the permissions on the logs.

Running a Splunk forwarder as root is somewhat common because of these permissions quandaries. It's simply an agent running locally which does not need to accept any incoming connections. (Everything a forwarder does is outbound initiated) Basically, the exposure of running a forwarder as root is relatively minimal. (Compared to the Splunk server which I would never recommend running as root) Yes, if Splunk is compromised the host is effectively rooted. But you have to be local to compromise Splunk, meaning you're likely rooted anyway.

There is also the fact that your application is writing logs to root:root 600 in the first place, which means your app runs as root. Just as much of a security risk as running Splunk as root, to be honest. That may not matter, but it's a precedence you can point to.

Highlighted

Re: How to get "splunk" user to read "root" user-owned files?

Influencer

One note to @emiller42's comment: Actually you would not necessarily have to be local to compromise a UF. Just like full blown Splunk it opens port 8089 for the REST API. Now there are options around firewalls, turning this off, and other protections that could be taken, but by itself, a UF running as root opens a service as root that assuming the existence of an exploit, could be compromised and root the box.