Hello having some confusing problems with Splunk permissions that I am trying to understand. Little background we upgrade our index/deployment server from Debian to ubuntu.
here is the problem I am seeing after this upgrade.
I was monitoring a file in var/log/test-combo.log and everything worked before hand on debian 11. Now I am not getting any of the data from this file ingested into my index but I can see fresh logs.
The file is owned by syslog and the group is adm.
My splunk user:
uid=1001(splunk) gid=1001(splunk) groups=1001(splunk),4(adm)
I wanted to do a test and I went under Data Inputs > Files & Directories > New Local File & Directory > Browse > Var > Log the strange thing was that I can see half of the logs and half of the directories under there. All the directories and files that I can root:root and had other: r-- set permissions the file in question (test-combo.log) didn't have other:r-- permissions set.
So why is splunk able to see files with these permissions
# file: vpn.log
# owner: root
# group: root
and not able to see files with this permission
# file: test-combo.log
# owner: syslog
# group: adm
is it because other is not set to read perms? What would be the significance of setting other to read?
As the splunkd process runs with a user which is a member of the adm group, it should be able to read the file as such.
But remember that in order to "reach" the file you need to have access to the directories containing the file (it's not a Novell Netware where when the leaf access was propagated as need "upstream" ;-)).
The easiest way to verify the permissions would be to su to the splunk user and try to read the file with cat or less.
Also check your input status with
splunk list inputstatus
and see what splunk has to say about this file.
BTW, you don't have SELinux enabled, do you?
@splunktrainingu - You may encountering the Splunk issue.
Run Splunk with the least privileged access on Linux - https://ideas.splunk.com/ideas/EID-I-1292
Please read the above idea description for details. Even though the idea status says "Under Point Threshold", I heard someone saying this has been resolved in Splunk 9.0.x. You can give it a try on a POC instance with the latest version of Splunk.
I hope this helps!!!
@splunktrainingu - Check below two things:
I hope this helps!!!
yes, due to permission issue splunk is not able to read the file,
as splunk user comes under under other user, you need give read permissions for able to read
also did did you see any permission related meesgaes in splunkd.log for test-combo.log source
If this reply helps you, Karma would be appreciated.
If your problem is resolved, then please click the "Accept as Solution" button to help future readers.
Thank you I checked the splunkd.log and found out it doesn't have permissions but I already knew that.
I am just trying to understand why? This doesn't make sense. Splunk user is part of ADM group, ADM group is applied to the file while can't splunk user read the file. What am I missing here?
Insufficient permissions to read file='/var/log/test-combo.log' (hint: Permission denied , UID: 1001, GID: 1001).