Why are cluster of indexers not getting data into custom index?

[irom77]

I have cluster of indexers i1, i2 and i3 and not seeing any data coming from universal forwarder f1 to custom index network. I can see index=_internal host="f1" on search head sh but nothing in network index.

I am filling up file random.log on f1

[ec2-user@f1 log]$ sudo /opt/splunkforwarder/bin/splunk btool inputs list monitor:///var/log/*.log
[monitor:///var/log/*.log]
_rcvbuf = 1572864
disabled = 0
host = $decideOnStartup
index = network

[ec2-user@f1 log]$ cat /var/log/random.log
Success 655
Error 78

Forwarder seems connected to Indexers

[ec2-user@f1 log]$ sudo tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
09-14-2022 12:59:15.389 +0000 INFO AutoLoadBalancedConnectionStrategy [2938 TcpOutEloop] - Connected to idx=10.0.7.4:9997, pset=0, reuse=0. using ACK.
09-14-2022 12:59:45.300 +0000 INFO AutoLoadBalancedConnectionStrategy [2938 TcpOutEloop] - Connected to idx=10.0.7.2:9997, pset=0, reuse=0. using ACK.
^C

[ec2-user@f1 log]$ sudo /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
10.0.7.2:9997
10.0.7.4:9997
Configured but inactive forwards:
10.0.7.3:9997

This is how it looks on one of indexers

[ec2-user@i1 ~]$ sudo /opt/splunk/bin/splunk list index | grep network
network
/opt/splunk/etc/network/db
/opt/splunk/etc/network/colddb
/opt/splunk/etc/network/thaweddb
[ec2-user@i1 ~]$ sudo ls -l /opt/splunk/etc/network/db
total 4
-rw------- 1 splunk splunk 10 Sep 14 11:45 CreationTime
drwx--x--- 2 splunk splunk 6 Sep 14 11:45 GlobalMetaData

[irom77 · Accepted solution]

Restarting UF did the thing...

[irom77]

Looks like I have correct rights (and running 8.2.6):

[ec2-user@f1 ~]$ getfacl /var/log/
getfacl: Removing leading '/' from absolute path names
# file: var/log/
# owner: root
# group: root
user::rwx
user:splunk:r--
group::r-x
mask::r-x
other::r-x

[ec2-user@f1 ~]$ getfacl /var/log/random.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/random.log
# owner: root
# group: root
user::rw-
user:splunk:r--
group::r--
mask::r--
other::r--

[irom77]

/var/log/ has default permissions.. So how I can monitor it without changing permissions on the dir ? 

[richgalloway]

Use the setfacl command to give the UF permission to read the file.

If you're running version 9+ then consider using its "Ambient Capabilities" to read any file.  See https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installleastprivileged

[mcxrisley08]

You can either run Splunk as root( not recommended) or change the permissions so that Splunk can read them.

[richgalloway]

Check the UF's internal logs for messages about being unable to read the monitored file.

[mcxrisley08]

What are the permissions on /var/log? This directory is not readable by anyone other than root by default.