Splunk Permission denied for log files

[apolloops]

We are using Splunk Enterprise server to send logs to be indexed. The monitor config is stored in '/opt/splunk/etc/system/local/inputs.conf'. An example monitor config is 

[monitor:///var/log/audit/audit.log]
sourcetype = linux_logs
index = splunk_server
disabled = false

But when splunk is restarted, the logs are not being sent and indexed. On looking the splunkd.log file, there was

WARN FilesystemChangeWatcher [9172 MainTailingThread] - error getting attributes of path "/var/log/aide/aide.log": Permission denied

this warning. Adding splunk user to root/sudo group didn't fix the issue. Note: We are using Splunk Enterprise server to send logs to itself through input.config file mentioned in this post (https://community.splunk.com/t5/Deployment-Architecture/Configure-a-Receiver-to-Forward-to-itself/m-...)

[PickleRick]

It's more your local OS admin's problem than Splunk issue as such.

In order to read the file, splunkd must have proper permissions to open the specified file(s) and read from it, which is quite obvious.

Usually it boils down to either running splunkd with a user/group which has access to all inputs it needs (although running it with root user might not be the best idea) or changing the permissions on files and directories splunkd has to read from (filesystem ACLs can be a good tool to finetune it if your filesystem supports them).

With linux you also might stumble upon selinux issues which can result in splunkd not being able to read files even though filesystem-level permissions seem to be OK. In that case you usually need to either relabel log files (but that can cause problems with the software that's writing to them) or create a selinux policy module which allows splunkd to read those files.