Question about inputs.conf [monitor:///...]

[verbal_666]

Hi there.

A simple question, it's not for a real usage, just a curiosity 😊

Does UF block inputs for system paths by default?

An example, teorically an inputs like this

[monitor:///...]
whitelist=.
index=root
sourcetype=root_all
disabled=0

Should ingest all non binary files under the "/" paths, including subdirs.

At the real fact, i find only the "/boot" path ingested.

Is this a security feature to exclude system paths "/" from been ingested?

Thanks 👍

[PickleRick]

I shouldn't think so. I'd expect it rather to be a permissions/SELinux issue or something like that.

Do

splunk list monitor

and

splunk inputstatus

[verbal_666]

Already done, System Paths are monitored, but no file is ingested 🙄

I think this is a security feature to exclude direct access to "/" 👍

Monitored Directories:
        /...
                /.autorelabel
                /afs
                /bin
                /boot
                /boot/.vmlinuz-5.14.0-284.11.1.el9_2.x86_64.hmac
                /boot/.vmlinuz-5.14.0-284.30.1.el9_2.x86_64.hmac
                /boot/config-5.14.0-284.11.1.el9_2.x86_64
                /boot/config-5.14.0-284.30.1.el9_2.x86_64
                /boot/efi
                /boot/grub2
                /boot/initramfs-0-rescue-d264ca908f764f5191a3c479f3e6f4bc.img
                /boot/initramfs-5.14.0-284.11.1.el9_2.x86_64.img
                /boot/initramfs-5.14.0-284.11.1.el9_2.x86_64kdump.img
                /boot/initramfs-5.14.0-284.30.1.el9_2.x86_64.img
                /boot/initramfs-5.14.0-284.30.1.el9_2.x86_64kdump.img
                /boot/loader
                /boot/symvers-5.14.0-284.11.1.el9_2.x86_64.gz
                /boot/symvers-5.14.0-284.30.1.el9_2.x86_64.gz
                /boot/System.map-5.14.0-284.11.1.el9_2.x86_64
                /boot/System.map-5.14.0-284.30.1.el9_2.x86_64
                /boot/vmlinuz-0-rescue-d264ca908f764f5191a3c479f3e6f4bc
                /boot/vmlinuz-5.14.0-284.11.1.el9_2.x86_64
                /boot/vmlinuz-5.14.0-284.30.1.el9_2.x86_64
                /dev
                /dev/almalinux
                /dev/block
                /dev/bsg
                /dev/cdrom
                /dev/char
                /dev/core
                /dev/cpu
                /dev/disk
                /dev/dma_heap
                /dev/dri
                /dev/fd
                /dev/hugepages
                /dev/initctl
                /dev/input
                /dev/log
                /dev/mapper
                /dev/mqueue
                /dev/net
                /dev/pts
                /dev/rtc
                /dev/shm
                /dev/snd
                /dev/stderr
                /dev/stdin
                /dev/stdout
                /dev/vfio
                /etc
                /home
                /lib
                /lib64
                /media
                /mnt
                /proc
                /proc/acpi
                /proc/bus
                /proc/dma
                /proc/fb
                /proc/fs
                /proc/irq
                /proc/keys
                /proc/kmsg
                /proc/net
                /proc/sys
                /proc/tty
                /root
                /run
                /sbin
                /srv
                /sys
                /This_is_Just_A_Test
                /usr

This can be also guessed by "/This_is_Just_A_Test" path, which contains many .txt files.

With "/..." they are skipped, with explicit,

[monitor:///This_is_Just_A_Test]

They are ingested 👍

I really think it's a security feature to prevent "/" to be fully accessed.

[PickleRick]

I did a test on my local UF.

It resulted with:

1) splunkd process opening hundreds of files (verifiable by "ls -la /proc/<splunk_pid>/fd")

2) Huge number of entries like:

03-09-2024 09:34:05.449 +0100 WARN FileClassifierManager [7610 tailreader0] - The file '/usr/bin/mariadb-import' is invalid. Reason: binary.
03-09-2024 09:34:05.449 +0100 INFO TailReader [7610 tailreader0] - Ignoring file '/usr/bin/mariadb-import' due to: binary

In splunkd.log

So splunkd is trying to read the files but doesn't ingest them due to the files being binary and then apparently gives up due to exhausting opened FDs limit (default - 100).

[verbal_666]

WARN .... Reason: binary.

Should be right, since no binary is grant in props and by default is set to not get access to binaries.

Descriptors to 100 is default, and it's ok, but should progress anyway. And in splunkd.log i can't see any WARN about descriptors.

Now, why the "/etc" with all its ascii system files are not ingested since it's before the "/usr"?

[PickleRick · Accepted solution]

The thing is that the file is being opened and is held open in case it's getting truncated and rewritten with textual contents. So the 100 fd limit is exhausted quickly.

About the order - I suppose either /bin is first (which in case of my Fedora is just a symlink to /usr/bin) or the order is the disk order not the alphabetical one.

[verbal_666]

I think it's a right analysis 👍

Maybe, just for some tests 😊i'll try playing with "max_fd" in limits to see how System works.

Just to stress the System 😂 as said in first post, it's only a test to understand better UF. I just saw how dangerous, sometimes, is to introduce the "..." or "*" or any other wildcard in path inputs, since UF could get crazy 🤷‍♂️like crcSalt, which could ingest x2/x3/x4/... data if not right blacklisted (think about log rotation with maybe gz/zip/bz extensions 🤷‍♂️ ).

Anyway, there's something else than only fd. In a stable environment, UF should "leave free" the file (and drop its fd) after "time_before_close" (5 by default), so can process other files in queue. Another strange situation, i can't see any WARN about fd in splunkd.log, as sometimes i saw in other situations 🤔🤔🤔log explicitly said the max_fd was raised 🤔now not! Strange!

Maybe this behaviour occurs different on different distros, as should be a System problem, not directly related to UF work 🤷‍♂️