Splunk file and directory monitoring- Am I configuring correctly?

[splunktrainingu]

Hello having some confusing problems with Splunk permissions that I am trying to understand. Little background we upgrade our index/deployment server from Debian to ubuntu.  

here is the problem I am seeing after this upgrade.

I was monitoring a file in var/log/test-combo.log  and everything worked before hand on debian 11. Now I am not getting any of the data from this file ingested into my index but I can see fresh logs.

The file is owned by syslog and the group is adm.

My splunk user:
uid=1001(splunk) gid=1001(splunk) groups=1001(splunk),4(adm)

I wanted to do a test and I went under Data Inputs > Files & Directories > New Local File & Directory > Browse > Var > Log the strange thing was that I can see half of the logs and half of the directories under there. All the directories and files that I can root:root and had other: r-- set permissions the file in question (test-combo.log) didn't have other:r-- permissions set. 

So why is splunk able to see files with these permissions

# file: vpn.log
# owner: root
# group: root
user::rw-
group::rw-
other::r--

and not able to see files with this permission

# file: test-combo.log
# owner: syslog
# group: adm
user::rw-
group::r--
other::---

is it because other is not set to read perms? What would be the significance of setting other to read?

[PickleRick]

As the splunkd process runs with a user which is a member of the adm group, it should be able to read the file as such.

But remember that in order to "reach" the file you need to have access to the directories containing the file (it's not a Novell Netware where when the leaf access was propagated as need "upstream" ;-)).

The easiest way to verify the permissions would be to su to the splunk user and try to read the file with cat or less.

Also check your input status with

splunk list inputstatus

and see what splunk has to say about this file.

BTW, you don't have SELinux enabled, do you?

[VatsalJagani]

Yes, interesting.

Yes, I would check file reading with the cat command with splunk user first.

[VatsalJagani]

@splunktrainingu - You may encountering the Splunk issue.

Run Splunk with the least privileged access on Linux -  https://ideas.splunk.com/ideas/EID-I-1292

Please read the above idea description for details. Even though the idea status says "Under Point Threshold", I heard someone saying this has been resolved in Splunk 9.0.x. You can give it a try on a POC instance with the latest version of Splunk.

I hope this helps!!!

[VatsalJagani]

@splunktrainingu - Check below two things:

• Make sure Splunk is running as the splunk user as you said.
  • ps -aux | grep "splunkd"
• Check for error logs in splunkd.log files.
  • index=_internal source="*splunkd.log*" error

I hope this helps!!!

[splunktrainingu]

splunk is running as the splunk user

[SanjayReddy]

Hi @splunktrainingu 

yes, due to permission issue splunk is not able to read the file,

as splunk user comes under under other user, you need give read permissions for able to read 

also did did you see any permission related meesgaes in splunkd.log for test-combo.log source

--------

Regards,
Sanjay Reddy

---
If this reply helps you, Karma would be appreciated.

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.

[splunktrainingu]

Thank you I checked the splunkd.log and found out it doesn't have permissions but I already knew that. 

I am just trying to understand why? This doesn't make sense. Splunk user is part of ADM group, ADM group is applied to the file while can't splunk user read the file. What am I missing here? 

Insufficient permissions to read file='/var/log/test-combo.log' (hint: Permission denied , UID: 1001, GID: 1001).