topic Re: FilesystemChangeWatcher - error getting attributes of path in Getting Data In

FilesystemChangeWatcher - error getting attributes of path?

nbowman — Wed, 22 Feb 2023 18:18:49 GMT

System specs:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.10 (Tikanga)
# uname -a
Linux llwbas1qa 2.6.18-371.9.1.el5 #1 SMP Tue May 13 06:52:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
# ./splunk version
Splunk Universal Forwarder 6.1.2 (build 213098)

I'm having an issue with one of my forwarders not forwarding properly. The files are being properly monitored for:

#./splunk list monitor
Monitored Directories:
        [No directories monitored.]
Monitored Files:
    /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/startServer.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log*

User splunk is part of the wasadmin group:

# cat /etc/passwd | grep 'wasadmin\|splunk'
wasadmin:x:650:650::/export/home/wasadmin:/bin/bash
splunk:x:502:1001:Splunk Server:/opt/splunkforwarder:/bin/bash
# cat /etc/group | grep wasadmin
wasadmin:x:650:splunk

When I restart splunk, I still get permission denied errors:

# /opt/splunkforwarder/var/log/splunk/splunkd.log
07-24-2014 17:04:41.650 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log*.
07-24-2014 17:04:41.650 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log*.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/startServer.log*.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Adding watch on path: /opt/IBM/WebSphere/wp_profile/ConfigEngine/log.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Adding watch on path: /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal.
07-24-2014 17:04:41.653 -0400 WARN  FilesystemChangeWatcher - error getting attributes of path "/opt/IBM/WebSphere/wp_profile/ConfigEngine/log": Permission denied
07-24-2014 17:04:41.653 -0400 WARN  FilesystemChangeWatcher - error getting attributes of path "/opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal": Permission denied

Permissions for user splunk (part of wasadmin group) to read the files seem fine:

# ls -la /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/ | grep log$
-rw-r--r-- 1 wasadmin wasadmin     167 Jul 24 09:05 native_stderr.log
-rw-r--r-- 1 wasadmin wasadmin    1758 Jul 24 09:05 native_stdout.log
-rw-r--r-- 1 wasadmin wasadmin    2034 Jul 24 09:10 startServer.log
-rw-r--r-- 1 wasadmin wasadmin  231382 Jul 25 08:51 SystemErr.log
-rw-r--r-- 1 wasadmin wasadmin  600072 Jul 25 11:10 SystemOut.log

# ls -la /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log 
-rw-r--r-- 1 wasadmin wasadmin 1481335 Mar 19 12:58 /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log

User splunk can read both files:

[splunk@hostname bin]$ tail -n 5 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log
    at com.ibm.ws.asynchbeans.AlarmImpl.runListenerAsCJWork(AlarmImpl.java:173)
    at com.ibm.ws.asynchbeans.am._Alarm.fireAlarm(_Alarm.java:332)
    at com.ibm.ws.asynchbeans.am._Alarm.run(_Alarm.java:229)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1560)

[splunk@hostname bin]$ tail -n 5 /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log 
     [echo] updated RegistrySynchronized in file wkplc.properties with value: true
Target finished: update-registry-sync-property

BUILD SUCCESSFUL
Total time: 43 seconds

SELinux is disabled:

# /usr/sbin/getenforce 
Disabled

What gives??

The only thing I can think of is that the log files are being locked out. lsof provides some insight:

# /usr/sbin/lsof /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/*
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
java    24001 wasadmin    0u   REG   8,17     1758 705832 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log
java    24001 wasadmin    1u   REG   8,17     1758 705832 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log
java    24001 wasadmin    2u   REG   8,17      167 705833 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log
java    24001 wasadmin    3u   REG   8,17  1959183 705836 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/verbosegc.20140724.090524.24001.txt.001
java    24001 wasadmin   26w   REG   8,17   599578 705838 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log
java    24001 wasadmin   27w   REG   8,17   231382 705839 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log

Has anyone ran into this before?

Re: FilesystemChangeWatcher - error getting attributes of path

tfpblanchard — Tue, 02 Dec 2014 18:44:52 GMT

Hi nbowman, I get the same error for similar settings, you're not alone 🙂

Re: FilesystemChangeWatcher - error getting attributes of path

segu — Fri, 09 Jan 2015 08:59:22 GMT

I had the same problem, did the same things as you did.
Then I updated the Splunk Universal Forwarder to 6.2.1 (build 245427) and the problem went away. Seems to be a bug.

Re: FilesystemChangeWatcher - error getting attributes of path

pgullette — Thu, 22 Jan 2015 20:16:23 GMT

I am beginning to see this happen as well in our enterprise. In our case, the splunk forwarders are deployed to Linux boxes and it's after a series of patches are applied to the boxes that the forwarder starts throwing these errors. My guess is that splunk is internally using some os command to get stats about a path, and whatever mechanism it's using is no longer allowed after the os patch. And I'd bet that 6.2 changes the method for stating a path which bypasses this behavior.

Maybe someone who works for Splunk will see this post and give us some insight.

Re: FilesystemChangeWatcher - error getting attributes of path

crash1011 — Fri, 18 Sep 2015 03:43:30 GMT

In case you get trapped with a file not being monitored even if (1) all permissions seem correct, (2) your deployment script is set to Enable App, Restart Splunkd and (3) You see these errors
09-18-2015 12:28:47.311 +1000 WARN FilesystemChangeWatcher - error getting attributes of path "/software/app/oracle/admin/webhost1/diagnostics/logs/OHS/ohs1/access_log": Permission denied
Then I found this actually did work:
- Log on to the forwarder and check that your app with the file monitoring stanza has been deployed all OK
- Do a splunk list monitor (if you’ve got the same problem it won’t be listed)
- Restart of splunk e.g. /opt/splunkforwarder/bin/splunk restart
- Do another splunk list monitor to see if it has worked

Unfortunately in this exercise I didn’t do a ps | grep splunk on the remote host to check if the splunkforwarder process had been restarted by the utility server’s splunk reload deploy-server

Re: FilesystemChangeWatcher - error getting attributes of path

Chris_Garrett — Wed, 10 May 2017 16:54:06 GMT

I just resolved this issue myself

TLDR: Any directories you're reading from, you must have read access to, and must have the execute bit set.

I highly recommend keeping selinux doing its job and executing the following for your hosts:

sudo setfacl -R -m u:splunk:rX /path/to/logs
The -R switch will apply permissions recursively
The -m is to modify the existing ACL
The u:splunk specifies the splunk user
The rX grants read access to everything, and sets the execute bit only on files with an existing execute bit flipped.

Cheers

Re: FilesystemChangeWatcher - error getting attributes of path

akocak — Fri, 22 Dec 2017 18:19:00 GMT

I appreciate all the answers here that I have used in my tasks. In addition to above, there is one more thing to check:

ACL entries of account that runs forwarder on RH (or SEL). If nothing works above, this is a good thing to check and add necessary config.

Re: FilesystemChangeWatcher - error getting attributes of path

splunkoptimus — Wed, 22 Feb 2023 12:37:16 GMT

I had the same error, I fixed it by giving splunk user read access to the logs and the directory they reside in.