Universal Forwarder not reading log files

[marcxbrl]

I'm having an problem where the universal forwarder isn't reading any log files except for syslog and messages. I've been looking at this issue for a while and I don't know where to look now.

When I set up the deployment server I organized the input files organized into a global file, web file, and server specific. Here's what they look like:

Global-inputs.conf

[monitor:///var/log/syslog*]
ignoreOlderThan=2d

[monitor:///var/log/messages*]
ignoreOlderThan=2d

[monitor:///var/log/custom/startup/*]
sourcetype=startuplogs
ignoreOlderThan=20d

[monitor:///var/log/custom/backup/*]
sourcetype=backuplogs
ignoreOlderThan=20d

web-inputs.conf

[monitor:///var/log/custom/apache2/*]
ignoreOlderThan=20d

server-input.conf

[monitor:///var/log/custom/report/report*]
sourcetype=report
ignoreOlderThan=20d

I started the forwarder, then made sure the configuration files were downloaded and applied correctly. The log file parses the monitors, but then they don't seem to analyze anything besides the first two sections in the global-inputs file.

Here's splunkd.log:

<snip>
08-10-2012 17:04:19.096 -0400 INFO  TailingProcessor - TailWatcher initializing...
08-10-2012 17:04:19.097 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/syslog*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/apache2/*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/backup/*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/report/report*.
08-10-2012 17:04:19.099 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/startup/*.
08-10-2012 17:04:19.099 -0400 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
08-10-2012 17:04:19.103 -0400 INFO  TcpOutputProc - Connected to idx=server_address:9578
08-10-2012 17:04:19.124 -0400 WARN  TailingProcessor - Insufficient permissions to read file='/opt/splunkforwarder/var/log/splunk/.splunkd.log.swp' (hint: Permission denied).
08-10-2012 17:04:19.126 -0400 INFO  ArchiveProcessor - handling file=/var/log/syslog.2.gz
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.126 -0400 INFO  ArchiveProcessor - reading path=/var/log/syslog.2.gz (seek=0 len=8676)
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.144 -0400 INFO  ArchiveProcessor - Finished processing file '/var/log/syslog.2.gz', removing from stats
</snip>

Nothing else is entered in the log for a good while after this. The metrics log continues to show connections to the main server.

I've made sure that the splunk user has the correct read permissions on the log files. I'm not getting bad permission errors. It seem to be skipping the other files completely. There's also entries in all the files newer than 20 days (limiting information during testing). The stateOnClient is enabled for each section in the serverclass.conf file.

What should I look for next?

[mslvrstn]

What about permissions on the /var/log/custom hierarchy?
Is it possible that the forwarder is not ingesting logs in there because the splunk user can't read them or search the containing directories?

[yannK]

The messages like "ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/messages[^/]$*" may not be relevant. see http://splunk-base.splunk.com/answers/47852/error-tailingprocessor-matching

To verify the monitored file lists, use the REST API on the forwarder, you will see if they are skipped and why :
https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

[marcxbrl]

Yes. I don't see any issues in the output. Here's a portion of the output:

global-inp [monitor:///var/log/custom/backup/]
system _rcvbuf = 1572864
system host = server_name
global-inp ignoreOlderThan = 10d
system index = test
global-inp sourcetype = backuplogs
global-inp [monitor:///var/log/custom/startup/]
system _rcvbuf = 1572864
system host = server_name
global-inp ignoreOlderThan = 10d
system index = test
global-inp sourcetype = startuplogs

[Lucas_K]

Can you see your inputs statement if you run btool?

ie. splunk cmd btool inputs list --debug

[marcxbrl]

I looked through the log but, looking at the global-input file only, it's not searching in the "...custom/startup/" or "...custom/backup/" directories. I don't see any reference to those directories in the output. It's like it's ignoring the second half of the config file.