How do I fix this weird permission case?

[danielbb]

We have a case where the data resides under /usr/feith/log/*.log and the Splunk process can read these files however, when I log in to the unix server I cannot navigate into this directory as the Splunk user. What's going on?

bash-4.4$ whoami
splunk
bash-4.4$ pwd
/usr/feith
bash-4.4$ \ls -tlr
total 388
...
drwxr-xr-x.    2 feith feith    4096 Dec 12 12:17 lib
drwx------.   19 feith feith    4096 Dec 13 01:00 log

bash-4.4$ cd log/
bash: cd: log/: Permission denied

[richgalloway]

Consider using the Least Privilege feature.  It allows a forwarder to read any file on the system.  See https://www.splunk.com/en_us/blog/learn/least-privilege-principle.html and https://docs.splunk.com/Documentation/Forwarder/9.3.2/Forwarder/Installleastprivileged

[PickleRick]

And what user does your splunkd run with?

[danielbb]

I see - 

$ ps -ef | grep splunk
splunk 2802446 2802413 0 Dec08 ? 00:00:08 [splunkd pid=2802413] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]

Meaning, the user splunk runs on the host and when I sudo to be the splunk user, I don't have access the these logs files, even though they are being ingested. 

[isoutamo]

You could check if there are additional ACL sets for that directory and especially for those files. Just make sudo to root (if possible) and then use getfacl command to look those https://www.computerhope.com/unix/ugetfacl.htm

How those file collections are defined in your inputs.conf? I think that with additional ACLs it's possible to define those so that you could read those files directly from that directory event you cannot cd into it.