Splunk Dev

How do I prevent Introspection Generator to read information about non splunk process when hidepid activated on /proc ?

maraman_splunk
Splunk Employee
Splunk Employee

Hi,

my splunk is running as splunk user on a linux system where the admin has secured the OS by using hidepid=1 on /proc (see https://ubuntuforums.org/showthread.php?t=2173093 and https://www.kernel.org/doc/Documentation/filesystems/proc.txt)

As a consequence, splunkd.log is filled with these error messages :
ERROR IntrospectionGenerator:resource_usage - RU - Fail to readlink(2) /proc/nnnn/exe: Operation not permitted where nnnn is a pid from a process not run by splunk
This is repeated for each pid so generate a lot of noise.

I would like to tell Introspection to only look at it's own pid in that case or not produce error message for this.

Any idea how to do this ?

0 Karma

ipfyx
Engager

Hi,

you can also add the splunk group gid to the fstab ($ id splunk_user) :
proc /proc proc rw,nosuid,nodev,noexec,relatime,gid=<splunk_gid>,hidepid=1 0 0

According to man proc :

       gid=gid (since Linux 3.3)
              Specifies the ID of a group whose members are authorized
              to learn process information otherwise prohibited by
              hidepid (i.e., users in this group behave as though /proc
              was mounted with hidepid=0).  This group should be used
              instead of approaches such as putting nonroot users into
              the sudoers(5) file.

 

maraman_splunk
Splunk Employee
Splunk Employee

As a workaround, I completely disabled the generator for resource usage
in server.conf

[introspection:generator:resource_usage]
disabled=true

this stop the error message flood but that will also disable all related stats in the monitoring console....

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...