Getting Data In

FilesystemChangeWatcher - error getting attributes of path

Path Finder

System specs:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.10 (Tikanga)
# uname -a
Linux llwbas1qa 2.6.18-371.9.1.el5 #1 SMP Tue May 13 06:52:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
# ./splunk version
Splunk Universal Forwarder 6.1.2 (build 213098)

I'm having an issue with one of my forwarders not forwarding properly. The files are being properly monitored for:

#./splunk list monitor
Monitored Directories:
        [No directories monitored.]
Monitored Files:
    /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/startServer.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log*
    /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log*

User splunk is part of the wasadmin group:

# cat /etc/passwd | grep 'wasadmin\|splunk'
wasadmin:x:650:650::/export/home/wasadmin:/bin/bash
splunk:x:502:1001:Splunk Server:/opt/splunkforwarder:/bin/bash
# cat /etc/group | grep wasadmin
wasadmin:x:650:splunk

When I restart splunk, I still get permission denied errors:

# /opt/splunkforwarder/var/log/splunk/splunkd.log
07-24-2014 17:04:41.650 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log*.
07-24-2014 17:04:41.650 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log*.
07-24-2014 17:04:41.651 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log*.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/startServer.log*.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Adding watch on path: /opt/IBM/WebSphere/wp_profile/ConfigEngine/log.
07-24-2014 17:04:41.652 -0400 INFO  TailingProcessor - Adding watch on path: /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal.
07-24-2014 17:04:41.653 -0400 WARN  FilesystemChangeWatcher - error getting attributes of path "/opt/IBM/WebSphere/wp_profile/ConfigEngine/log": Permission denied
07-24-2014 17:04:41.653 -0400 WARN  FilesystemChangeWatcher - error getting attributes of path "/opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal": Permission denied

Permissions for user splunk (part of wasadmin group) to read the files seem fine:

# ls -la /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/ | grep log$
-rw-r--r-- 1 wasadmin wasadmin     167 Jul 24 09:05 native_stderr.log
-rw-r--r-- 1 wasadmin wasadmin    1758 Jul 24 09:05 native_stdout.log
-rw-r--r-- 1 wasadmin wasadmin    2034 Jul 24 09:10 startServer.log
-rw-r--r-- 1 wasadmin wasadmin  231382 Jul 25 08:51 SystemErr.log
-rw-r--r-- 1 wasadmin wasadmin  600072 Jul 25 11:10 SystemOut.log

# ls -la /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log 
-rw-r--r-- 1 wasadmin wasadmin 1481335 Mar 19 12:58 /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log

User splunk can read both files:

[splunk@hostname bin]$ tail -n 5 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log
    at com.ibm.ws.asynchbeans.AlarmImpl.runListenerAsCJWork(AlarmImpl.java:173)
    at com.ibm.ws.asynchbeans.am._Alarm.fireAlarm(_Alarm.java:332)
    at com.ibm.ws.asynchbeans.am._Alarm.run(_Alarm.java:229)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1560)

[splunk@hostname bin]$ tail -n 5 /opt/IBM/WebSphere/wp_profile/ConfigEngine/log/ConfigTrace.log 
     [echo] updated RegistrySynchronized in file wkplc.properties with value: true
Target finished: update-registry-sync-property

BUILD SUCCESSFUL
Total time: 43 seconds

SELinux is disabled:

# /usr/sbin/getenforce 
Disabled

What gives??

The only thing I can think of is that the log files are being locked out. lsof provides some insight:

# /usr/sbin/lsof /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/*
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
java    24001 wasadmin    0u   REG   8,17     1758 705832 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log
java    24001 wasadmin    1u   REG   8,17     1758 705832 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stdout.log
java    24001 wasadmin    2u   REG   8,17      167 705833 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/native_stderr.log
java    24001 wasadmin    3u   REG   8,17  1959183 705836 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/verbosegc.20140724.090524.24001.txt.001
java    24001 wasadmin   26w   REG   8,17   599578 705838 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log
java    24001 wasadmin   27w   REG   8,17   231382 705839 /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemErr.log

Has anyone ran into this before?

Contributor

I appreciate all the answers here that I have used in my tasks. In addition to above, there is one more thing to check:

ACL entries of account that runs forwarder on RH (or SEL). If nothing works above, this is a good thing to check and add necessary config.

0 Karma

I just resolved this issue myself

TLDR: Any directories you're reading from, you must have read access to, and must have the execute bit set.

I highly recommend keeping selinux doing its job and executing the following for your hosts:

sudo setfacl -R -m u:splunk:rX /path/to/logs
The -R switch will apply permissions recursively
The -m is to modify the existing ACL
The u:splunk specifies the splunk user
The rX grants read access to everything, and sets the execute bit only on files with an existing execute bit flipped.

Cheers

Explorer

In case you get trapped with a file not being monitored even if (1) all permissions seem correct, (2) your deployment script is set to Enable App, Restart Splunkd and (3) You see these errors
09-18-2015 12:28:47.311 +1000 WARN FilesystemChangeWatcher - error getting attributes of path "/software/app/oracle/admin/webhost1/diagnostics/logs/OHS/ohs1/access_log": Permission denied
Then I found this actually did work:
- Log on to the forwarder and check that your app with the file monitoring stanza has been deployed all OK
- Do a splunk list monitor (if you’ve got the same problem it won’t be listed)
- Restart of splunk e.g. /opt/splunkforwarder/bin/splunk restart
- Do another splunk list monitor to see if it has worked

Unfortunately in this exercise I didn’t do a ps | grep splunk on the remote host to check if the splunkforwarder process had been restarted by the utility server’s splunk reload deploy-server

Explorer

I had the same problem, did the same things as you did.
Then I updated the Splunk Universal Forwarder to 6.2.1 (build 245427) and the problem went away. Seems to be a bug.

Explorer

I am beginning to see this happen as well in our enterprise. In our case, the splunk forwarders are deployed to Linux boxes and it's after a series of patches are applied to the boxes that the forwarder starts throwing these errors. My guess is that splunk is internally using some os command to get stats about a path, and whatever mechanism it's using is no longer allowed after the os patch. And I'd bet that 6.2 changes the method for stating a path which bypasses this behavior.

Maybe someone who works for Splunk will see this post and give us some insight.

0 Karma

Explorer

Hi nbowman, I get the same error for similar settings, you're not alone 🙂

0 Karma