We are running the Splunkforwarder as user splunk and group splunk on our Solaris and AIX systems. We then give Splunk access to the log files either by putting ACLs on this files, or by changing the group to Splunk and give group read access. It gets more tricky for apps with scripted inputs, usually we run a cron job outside of Splunk, e.g as the oracle user, which runs the scripts and writes its output to a file which is in turn read by Splunk.
I too am interested in knowing a technique for having Splunk Forwarder installed and running as non-root but still able to monitor root read only /var/adm/s* log files. Do you have a technique or a special Splunk privileged module or a startup parameter to enable reading root files?
In non-root mode, nothing in /opt/splunkforwarder is root owned, but do you have a special module that could be root owned to enable the reading of root only logs? I have to deploy SF to a couple of dozen servers and don't want to have them all running as root for a single log file on each server.
The following technique will work: 1) add a new special group; 2) adding SF to that group and only having SF in that group; 3) changing the group of the file with 600 permission to the new special group; 4) change the group access permission to add R to the file so that it is 640. This means that the file will be group readable by SF but since SF is the only userid in that group, only SF can read the file.
This will work so long as the permissions and group ownership do not change. In my case, the initial test worked, but the file got reverted, so this is not a workable solution.
I am still left with SF as non-root but unable to read root 600 files.
I think the Splunk Forwarder product should come with a special read module that can be rooted so that it can read root 600 files even though the rest of SF is installed non-root.
You can have splunk run as any user you please (see here http://docs.splunk.com/Documentation/Splunk/6.3.3/Installation/RunSplunkasadifferentornon-rootuser) but of course it will only be able to monitor files to which it has read access.
For example /var/log/syslog is usually only readable by root. if you have a non root user that can read the files you want go for it!
Alternatively I would suggest leaving splunk running under root but restrict incoming traffic to the management port to your deployment server, or if you don't use a deployment server restrict the management port to localhost only.
Sounds pretty counter-intuitive to me, but thanks for the response.
Could somebody from Splunk Engineering chime in with their recommendation please ? I keep finding these edge cases where simple stuff is difficult/impossible/too-complicated in this product. Other than "don't do that" it would be nice to have a "do this instead" from Splunk Engineering 'officially' as a best practice. You know we will want to monitor our splunk server logs just like any other forwarding computer. What is the way we are supposed to do this while meeting your recommendation to not run the server processes as root ?
Well, this is centos, so it's not adm, and many interesting files are mode 600 so group permissions wouldn't help. I'm reluctant to hack up the system to force things. Good idea though...
yes, on centos it's root:root - I know 'how' to hack the system up, I just don't want to do it.
Still awaiting an official Splunk chime-in on what they want us to do. Their silence on these kinds of things is a bit disconcerting.
On a similar note - I know that Splunk recommends not running a 'server' (not forwarder) as root, but then it can't monitor itself for the same reason. I set up a new server to run as user splunk group splunk and couldn't monitor /var/log/messages due to permission denied.
Is there any reasonable way to let the non-privileged splunk processes monitor non-splunk-owned files ? Do I need to do something crazy like forwarding syslog to the indexer rather than using splunk ? Seems dumb when you get to that level of complexity when I can packet filter things and block input to the whole server.
Ideas ? Best practices that actually work ?
Well the spunk server needs to be accessible remotely hence the need to run it as a non root user. Just have a separate Universal Forwarder instance on your spunk server that runs as root and does not listen to external connections.