Getting Data In

Splunk file and directory monitoring- Am I configuring correctly?

splunktrainingu
Communicator

Hello having some confusing problems with Splunk permissions that I am trying to understand. Little background we upgrade our index/deployment server from Debian to ubuntu.  

here is the problem I am seeing after this upgrade.

 

I was monitoring a file in var/log/test-combo.log  and everything worked before hand on debian 11. Now I am not getting any of the data from this file ingested into my index but I can see fresh logs.

The file is owned by syslog and the group is adm.

My splunk user:
uid=1001(splunk) gid=1001(splunk) groups=1001(splunk),4(adm)

I wanted to do a test and I went under Data Inputs > Files & Directories > New Local File & Directory > Browse > Var > Log the strange thing was that I can see half of the logs and half of the directories under there. All the directories and files that I can root:root and had other: r-- set permissions the file in question (test-combo.log) didn't have other:r-- permissions set. 

So why is splunk able to see files with these permissions

# file: vpn.log
# owner: root
# group: root
user::rw-
group::rw-
other::r--

 

and not able to see files with this permission

 

# file: test-combo.log
# owner: syslog
# group: adm
user::rw-
group::r--
other::---

is it because other is not set to read perms? What would be the significance of setting other to read?

Labels (1)
0 Karma

PickleRick
Ultra Champion

As the splunkd process runs with a user which is a member of the adm group, it should be able to read the file as such.

But remember that in order to "reach" the file you need to have access to the directories containing the file (it's not a Novell Netware where when the leaf access was propagated as need "upstream" ;-)).

The easiest way to verify the permissions would be to su to the splunk user and try to read the file with cat or less.

Also check your input status with

splunk list inputstatus

and see what splunk has to say about this file.

BTW, you don't have SELinux enabled, do you?

0 Karma

VatsalJagani
Super Champion

Yes, interesting.

Yes, I would check file reading with the cat command with splunk user first.

0 Karma

VatsalJagani
Super Champion

@splunktrainingu - You may encountering the Splunk issue.

Run Splunk with the least privileged access on Linux -  https://ideas.splunk.com/ideas/EID-I-1292

Please read the above idea description for details. Even though the idea status says "Under Point Threshold", I heard someone saying this has been resolved in Splunk 9.0.x. You can give it a try on a POC instance with the latest version of Splunk.

 

I hope this helps!!!

0 Karma

VatsalJagani
Super Champion

@splunktrainingu - Check below two things:

  • Make sure Splunk is running as the splunk user as you said.
    • ps -aux | grep "splunkd"
  • Check for error logs in splunkd.log files.
    • index=_internal source="*splunkd.log*" error

 

I hope this helps!!!

0 Karma

splunktrainingu
Communicator

splunk is running as the splunk user

0 Karma

SanjayReddy
Builder

Hi @splunktrainingu 

yes, due to permission issue splunk is not able to read the file,

as splunk user comes under under other user, you need give read permissions for able to read 

also did did you see any permission related meesgaes in splunkd.log for test-combo.log source

--------

Regards,
Sanjay Reddy

---
If this reply helps you, Karma would be appreciated.

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.

 

----
Regards,
Sanjay Reddy

----
If this reply helps you, Karma would be appreciated.
0 Karma

splunktrainingu
Communicator

Thank you I checked the splunkd.log and found out it doesn't have permissions but I already knew that. 

I am just trying to understand why? This doesn't make sense. Splunk user is part of ADM group, ADM group is applied to the file while can't splunk user read the file. What am I missing here? 

 

Insufficient permissions to read file='/var/log/test-combo.log' (hint: Permission denied , UID: 1001, GID: 1001).

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...