Security
Highlighted

How to monitor root owned logs while running Splunk as a non-root user

Engager

The Splunk indexer and forwarders in my environment are configured to run as the "splunk" user for security reasons. Of course, this means that Splunk can no longer read root owned log files. The first two thoughts to cross my mind were to either use filesystem ACLs to provide read access to the splunk user or employ the use of a dedicated usergroup.

I'm just curious to see what people in this type of environment are doing to get around this issue? Have you run into any specific issues one way or the other?

Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Path Finder

If you're using Solaris 10, the fine grained privileges model makes this really easy. In your SMF manifest, do this:

    <method_context project=':default' resource_pool=':default' working_directory=':default'>
  <method_credential group='splunk' limit_privileges=':default' privileges='basic,file_dac_read,net_privaddr' supp_groups=':default' user='splunk'/>
  <method_environment>
    <envvar name='HOME' value='/opt/splunk'/>
  </method_environment>
</method_context>

Giving the service the service the net_privaddr privilege allows it to bind to low ports without ever having been root, and file_dac_read allows it to read other users' files that it would not otherwise be able to read.

If you're on an inferior operating system like linux, there will be significantly more klugding, I would think 😕

Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

New Member

I have noticed that when I install the *nix app, I am still receiving audit.log and other entries that are usually owned by root. How are we doing this? We would like to run as splunk on indexers and search heads but the concern is that we will no longer be able to obtain logs owned by root. What is the solution here?

0 Karma
Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Super Champion

Good question.

I'm running on the "inferior" Linux platform... 😉

I've generally found the easiest approach is to add splunk to my local admin group--the only group of users who are allowed to see the log files (besides root, or whoever actually owns the log file.) This may not be the "best" approach, but it keeps splunk only having read-only access to the logs, and doesn't require making log files world-readable.

We use mostly syslog-ng internally. We have syslog-ng write everything to local files, and then use splunk forwarders to read the log files and forward the events to our central indexer. (If all the logs went through syslog, then we may consider just doing the syslog forwarding thing, but there always seems to be one or two apps that don't use syslog. Therefore we need a splunk forwarder anyways, so it's just easier to have the splunk forwarder pickup all the log files.)

We stick the following file permissions settings in our syslog-ng.conf file:

options {
    ...
    owner(root);
    group(adm);
    perm(0640);
    ...
};

Then for any logs that rely on logrotate for creation, I use settings like this:

/var/log/log_file_name {
    ...
    create 0640 root adm
    delaycompress
    olddir old
    dateext
    ....

}

I recommend the delaycompress setting, just in case splunk is still reading events from the file at the point that it is being rotated (which isn't super likely, and may be handled automatically in any case, but it makes me feel better). The delay option also lets you index not only the primary log file but the previously rotated one as well. (Splunk will detect that it was rotated and will continue reading it from where it left off -- just in case any events remained that weren't indexed yet. This is not usually a problem, but say splunkd was restarted just a few seconds before the log rotation kicked off, you could end up with a few missing events on some busy log files.)

I also like sticking the old logs in a separate directory using olddir, which lets me keep more history but not clutter up the main log directory (although since we now use splunk to store them for much longer, we may reconsider this and simply delete them more frequently. I don't know, but it's something to think about. At least until you get comfortable with splunk, you may want to keep your existing retention policy, just in case...) Either way, you need to make sure splunk knows not to try to read the compressed version of your rotated log files. Normally splunk will detect rotated log files and not re-process them, but if you rotate and then compress, splunk doesn't recognize that your logfile.20100629.gz file is the same file as logfile from two days ago, and therefore your events are loaded twice (once in the original log file, and then once with the compressed file.) There are a number of ways to avoid this problem this, but it could be easily overlooked.

I like the dateext option too. This is especially helpful if you have to go back and reload log files from a previous year. (Splunk will pickup the timestamp from the filename and therefore get the year correct. Since the default syslog timestamp format doesn't include the year, splunk has to guess, and can often get it wrong, unfortunately.) This also make it easier on any incremental backup tools since you don't keep renumbering the same file over and over. But generally speaking, splunk will handle either well automatically.


I do have an older linux system with an older version syslog-ng that doesn't support setting the file mode and group, so I had to make a simple cron job that goes around and does a bumch of chmod and chgrp commands to reset the permissions after a log rotation. I feel like this is sloppy, but its' the best solution I could come up with on that system.

And of course, the risk of running splunk as root is that is can easily index file you don't want it to. (For example /etc/passwd or /etc/shadow, which is a known issue by the way...) Not to mention any kind of security vulnerability that could let a remote use take over splunk and easily gain root access.

If you do have the need to have splunk run saved searches that executes scripts, you can always setup sudo to allow to run very specific commands as root without requiring a password. In my opinion, this is a much safer approach than running splunk as root and hoping that it doesn't execute something that it shouldn't. And setting up sudo in this way isn't all that difficult either.

Also in terms of security, make sure you change the splunk admin password. Even on your forwarders where splunkweb isn't running. You can remote control your splunkd process over port 8089. (Simply open a browser to https://your.splunk.forwarder:8089/services to see what I mean.) If you don't want to do any remote splunkd admin kind of stuff, then you may also want to block that port with a local firewall, but changing that password should be done in either case.

Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Communicator

For (current in 2018) reference - Regarding port 8089 on Universal Forwarders:

In server.conf use:

[httpServer]
disableDefaultPort = true

We set this in a special deployment app for all universal forwarders.

0 Karma
Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Path Finder

On Redhat we run splunk in its own user/group. Then add read only ACL permissions to the splunk group to specific files.
You can manually set the ACL with

sudo setfacl -m g:splunk:rx /var/log/messages

This will not persist as logrotate will not re-apply the ACL setting so for a more permanent solution we have added a rule to logrotate to reset the ACL.
we added the file..

/etc/logrotate.d/Splunk_ACLs
with
{
postrotate
/usr/bin/setfacl -m g:splunk:rx /var/log/cron
/usr/bin/setfacl -m g:splunk:rx /var/log/maillog
/usr/bin/setfacl -m g:splunk:rx /var/log/messages
/usr/bin/setfacl -m g:splunk:rx /var/log/secure
/usr/bin/setfacl -m g:splunk:rx /var/log/spooler
endscript
}

Check the ACL status of a file with

$ getfacl /var/log/messages

Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Path Finder

for more info on ACLs take a look at https://help.ubuntu.com/community/FilePermissionsACLs

0 Karma
Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Path Finder
0 Karma
Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

I just get the error lines must begin with a keyword or a filename (possibly in double quotes) with this kind of configuration.

Highlighted

Re: How to monitor root owned logs while running Splunk as a non-root user

Communicator

As written below for the other answers on setfacl: do not set the x-bit on the files. They are not (or should not be) executed.

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.