I would like to know, how security conscious teams configure (like guid, user account to run splunkd as, etc for) Splunk Universal Forwarder on *nix platform, to monitor both the privileged (logs like /var/log/message with perm 600) and non-privileged (may be an application log with perm 666).
I looked at the below answer and don't think installing syslog-ng on all servers is a option for us.
http://answers.splunk.com/answers/4253/how-to-monitor-root-owned-logs-while-running-splunk-as-a-non-...
Due to how the UF operates, Linux capabilities are not an option. The access system call will incorrectly claim that the splunkd cannot open the file, when an actual attempt to open it for reading would succeed, so splunkd never tries.
Some of us have found this problem and spent time confirming the issue.
I've used setfacl, though that is a bit fragile at times. If you only have a very fixed number of files, you could write a configuration management policy to ensure file acls and default file acls are set for the appropriate files and logs. If you do use file ACLs, make sure and set the default file ACLs on the directory, and set log rotation parameters appropriately (default ACLs protect you from most log rotation issues, unless you create new directories).