How do I get Monitoring for JSON log files in docker host volume mount directories with Splunk Universal Forwarder working? Getting permission issues

[shodudley]

I'm working with Splunk Universal Forwarder 6.5.2 and am trying to configure a monitor on the docker volumes directory on docker hosts.

I have a app configuration "inputs.conf" like this:

[default]
host=THEHOSTNAMEHERE

[monitor:///var/lib/docker/volumes/splunk-log-store/.*]
sourcetype=json
index=application

In the splunk-log-store directory I have a number of .log files with JSON content.

[root@THEHOSTNAMEHERE ec2-user]# tail -n 100 $SPLUNK_HOME/var/log/splunk/splunkd.log | grep -E 'permission|docker'

05-02-2019 14:19:35.718 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/lib/docker/volumes/splunk-log-store/.*.
05-02-2019 14:19:35.719 +0000 INFO  TailingProcessor - Adding watch on path: /var/lib/docker/volumes/splunk-log-store.
05-02-2019 14:19:35.719 +0000 WARN  FilesystemChangeWatcher - error getting attributes of path "/var/lib/docker/volumes/splunk-log-store": Permission denied
05-02-2019 14:19:35.721 +0000 WARN  TailReader - Insufficient permissions to read file='/var/log/cron' (hint: Permission denied ,                            UID: 501, GID: 501).
05-02-2019 14:19:35.722 +0000 WARN  TailReader - Insufficient permissions to read file='/var/log/secure' (hint: Permission denied ,                            UID: 501, GID: 501).
05-02-2019 14:19:35.722 +0000 WARN  TailReader - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied ,                            UID: 501, GID: 501).

However, as per above, I'm getting permission errors when the splunk service starts.

It's running as user "splunk" (UID 501).

This is the particular error around the directory I'm trying to monitor:

WARN FilesystemChangeWatcher - error getting attributes of path "/var/lib/docker/volumes/splunk-log-store": Permission denied

I've tried changing the group permissions on the directory to the splunk group and also tried setting full permissions (completely open) as a hacky test but each time I restart the splunk service I still get this permission error.

Any ideas what I'm doing wrong here?

I have been able to successfully get syslog logs working by modifying permissions on the /var/log/messages etc... but the same modifications for this docker volume directory just don't seem to work.

Thanks