Universal Forwarder not reading log files

marcxbrl — Fri, 10 Aug 2012 22:04:06 GMT

I'm having an problem where the universal forwarder isn't reading any log files except for syslog and messages. I've been looking at this issue for a while and I don't know where to look now.

When I set up the deployment server I organized the input files organized into a global file, web file, and server specific. Here's what they look like:

Global-inputs.conf

[monitor:///var/log/syslog*]
ignoreOlderThan=2d

[monitor:///var/log/messages*]
ignoreOlderThan=2d

[monitor:///var/log/custom/startup/*]
sourcetype=startuplogs
ignoreOlderThan=20d

[monitor:///var/log/custom/backup/*]
sourcetype=backuplogs
ignoreOlderThan=20d

web-inputs.conf

[monitor:///var/log/custom/apache2/*]
ignoreOlderThan=20d

server-input.conf

[monitor:///var/log/custom/report/report*]
sourcetype=report
ignoreOlderThan=20d

I started the forwarder, then made sure the configuration files were downloaded and applied correctly. The log file parses the monitors, but then they don't seem to analyze anything besides the first two sections in the global-inputs file.

Here's splunkd.log:

<snip>
08-10-2012 17:04:19.096 -0400 INFO  TailingProcessor - TailWatcher initializing...
08-10-2012 17:04:19.097 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/syslog*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/apache2/*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/backup/*.
08-10-2012 17:04:19.098 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/report/report*.
08-10-2012 17:04:19.099 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/startup/*.
08-10-2012 17:04:19.099 -0400 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
08-10-2012 17:04:19.103 -0400 INFO  TcpOutputProc - Connected to idx=server_address:9578
08-10-2012 17:04:19.124 -0400 WARN  TailingProcessor - Insufficient permissions to read file='/opt/splunkforwarder/var/log/splunk/.splunkd.log.swp' (hint: Permission denied).
08-10-2012 17:04:19.126 -0400 INFO  ArchiveProcessor - handling file=/var/log/syslog.2.gz
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.126 -0400 INFO  ArchiveProcessor - reading path=/var/log/syslog.2.gz (seek=0 len=8676)
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.144 -0400 INFO  ArchiveProcessor - Finished processing file '/var/log/syslog.2.gz', removing from stats
</snip>

Nothing else is entered in the log for a good while after this. The metrics log continues to show connections to the main server.

I've made sure that the splunk user has the correct read permissions on the log files. I'm not getting bad permission errors. It seem to be skipping the other files completely. There's also entries in all the files newer than 20 days (limiting information during testing). The stateOnClient is enabled for each section in the serverclass.conf file.

What should I look for next?

Re: Universal Forwarder not reading log files

yannK — Sat, 11 Aug 2012 06:56:59 GMT

The messages like "ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/messages[^/]$*" may not be relevant. see http://splunk-base.splunk.com/answers/47852/error-tailingprocessor-matching

To verify the monitored file lists, use the REST API on the forwarder, you will see if they are skipped and why :
https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Re: Universal Forwarder not reading log files

marcxbrl — Sun, 12 Aug 2012 03:24:07 GMT

I looked through the log but, looking at the global-input file only, it's not searching in the "...custom/startup/" or "...custom/backup/" directories. I don't see any reference to those directories in the output. It's like it's ignoring the second half of the config file.

Re: Universal Forwarder not reading log files

Lucas_K — Mon, 13 Aug 2012 06:01:34 GMT

Can you see your inputs statement if you run btool?

ie. splunk cmd btool inputs list --debug

Re: Universal Forwarder not reading log files

marcxbrl — Mon, 28 Sep 2020 12:15:17 GMT

Yes. I don't see any issues in the output. Here's a portion of the output:

global-inp [monitor:///var/log/custom/backup/]
system _rcvbuf = 1572864
system host = server_name
global-inp ignoreOlderThan = 10d
system index = test
global-inp sourcetype = backuplogs
global-inp [monitor:///var/log/custom/startup/]
system _rcvbuf = 1572864
system host = server_name
global-inp ignoreOlderThan = 10d
system index = test
global-inp sourcetype = startuplogs

Re: Universal Forwarder not reading log files

mslvrstn — Wed, 15 Aug 2012 05:07:33 GMT

What about permissions on the /var/log/custom hierarchy?
Is it possible that the forwarder is not ingesting logs in there because the splunk user can't read them or search the containing directories?

topic Re: Universal Forwarder not reading log files in Getting Data In

Universal Forwarder not reading log files

Re: Universal Forwarder not reading log files

Re: Universal Forwarder not reading log files

Re: Universal Forwarder not reading log files

Re: Universal Forwarder not reading log files

Re: Universal Forwarder not reading log files