Getting Data In

Why are cluster of indexers not getting data into custom index?

irom77
Explorer

I have cluster of indexers i1, i2 and i3 and not seeing any data coming from universal forwarder f1 to custom index network. I can see index=_internal host="f1" on search head sh but nothing in network index.

I am filling up file random.log on f1

[ec2-user@f1 log]$ sudo /opt/splunkforwarder/bin/splunk btool inputs list monitor:///var/log/*.log
[monitor:///var/log/*.log]
_rcvbuf = 1572864
disabled = 0
host = $decideOnStartup
index = network

[ec2-user@f1 log]$ cat /var/log/random.log
Success 655
Error 78

Forwarder seems connected to Indexers

[ec2-user@f1 log]$ sudo tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
09-14-2022 12:59:15.389 +0000 INFO AutoLoadBalancedConnectionStrategy [2938 TcpOutEloop] - Connected to idx=10.0.7.4:9997, pset=0, reuse=0. using ACK.
09-14-2022 12:59:45.300 +0000 INFO AutoLoadBalancedConnectionStrategy [2938 TcpOutEloop] - Connected to idx=10.0.7.2:9997, pset=0, reuse=0. using ACK.
^C

[ec2-user@f1 log]$ sudo /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
10.0.7.2:9997
10.0.7.4:9997
Configured but inactive forwards:
10.0.7.3:9997

This is how it looks on one of indexers

[ec2-user@i1 ~]$ sudo /opt/splunk/bin/splunk list index | grep network
network
/opt/splunk/etc/network/db
/opt/splunk/etc/network/colddb
/opt/splunk/etc/network/thaweddb
[ec2-user@i1 ~]$ sudo ls -l /opt/splunk/etc/network/db
total 4
-rw------- 1 splunk splunk 10 Sep 14 11:45 CreationTime
drwx--x--- 2 splunk splunk 6 Sep 14 11:45 GlobalMetaData

Labels (1)
0 Karma
1 Solution

irom77
Explorer

Restarting UF did the thing...

View solution in original post

irom77
Explorer

Restarting UF did the thing...

irom77
Explorer

Looks like I have correct rights (and running 8.2.6):

 

[ec2-user@f1 ~]$ getfacl /var/log/
getfacl: Removing leading '/' from absolute path names
# file: var/log/
# owner: root
# group: root
user::rwx
user:splunk:r--
group::r-x
mask::r-x
other::r-x

[ec2-user@f1 ~]$ getfacl /var/log/random.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/random.log
# owner: root
# group: root
user::rw-
user:splunk:r--
group::r--
mask::r--
other::r--

 

0 Karma

irom77
Explorer

/var/log/ has default permissions.. So how I can monitor it without changing permissions on the dir ? 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Use the setfacl command to give the UF permission to read the file.

If you're running version 9+ then consider using its "Ambient Capabilities" to read any file.  See https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installleastprivileged

---
If this reply helps you, Karma would be appreciated.

mcxrisley08
Path Finder

You can either run Splunk as root( not recommended) or change the permissions so that Splunk can read them.

richgalloway
SplunkTrust
SplunkTrust

Check the UF's internal logs for messages about being unable to read the monitored file.

---
If this reply helps you, Karma would be appreciated.

mcxrisley08
Path Finder

What are the permissions on /var/log? This directory is not readable by anyone other than root by default.

Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...