Getting Data In

Why isn't splunkd.log getting forwarded (in 4.0.x, fixed in 4.1) ?

Communicator

UPDATE: This appears to be a bug specifically related to 4.0.10. The following is a work around in system/local/inputs.conf

[monitor:///home/ops/splunk/var/log/splunk]
disabled=true


It looks like the entry in system/default is clobbering the more granular entries in SplunkLightForwarder/default. The output of splunk list monitor seems to be broken as well

I'd like to index the splunkd.log for remote troubleshooting but I can't get my light forwarders to forward the log. Here's the btool output (with unrelated stuff removed):

[root@neil bin]# ./splunk cmd btool --debug inputs list
system     [default]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = default
system     [monitor:///home/ops/splunk/var/log/splunk]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = _internal
SplunkLigh [monitor:///home/ops/splunk/var/log/splunk/splunkd.log]
SplunkLigh _TCP_ROUTING = *
system     _rcvbuf = 1572864
system     host = myforwarder
maint      index = _internal
maint      sourcetype = splunkd
atti-linux [monitor:///var/log]
system     _rcvbuf = 1572864
atti-linux _whitelist = (auth$|cron$|kern$|lpr$|maillog$|user$|local$)
system     host = myforwarder
system     index = default
system     [splunktcp]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = default
system     route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

I tried changing the index, no dice. From what I can tell, the /var/log/splunk entry clobbers the more specific one which says to include the entire var/log/splunk directory. When I run "splunk list monitor" it prints out the entire var/log/splunk dir except splunkd.log!

[root@neil-search etc]# ../bin/splunk list monitor
Monitored Directories:
    $SPLUNK_HOME/var/log/splunk
        /home/ops/splunk/var/log/splunk/audit.log
        /home/ops/splunk/var/log/splunk/btool.log
        /home/ops/splunk/var/log/splunk/metrics.log.1
        /home/ops/splunk/var/log/splunk/metrics.log.2
        /home/ops/splunk/var/log/splunk/metrics.log.3
        /home/ops/splunk/var/log/splunk/metrics.log.4
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-23.00-14-09
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-23.17-02-12
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-26.18-58-16
        /home/ops/splunk/var/log/splunk/searchhistory.log
        /home/ops/splunk/var/log/splunk/splunkd_access.log
        /home/ops/splunk/var/log/splunk/splunkd_stderr.log
        /home/ops/splunk/var/log/splunk/splunkd_stdout.log
        /home/ops/splunk/var/log/splunk/splunklogger.log
Tags (2)
1 Solution

Champion

For clarification, this is applies to Lightweight Forwarder. The default expected behavior (LWF forwards its internal logs as default) is not working in version 4.0.x. The workaround is to create a monitor input for the Splunk logs directory and whitelist specific files in the $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *
_whitelist = (splunkd|metrics|license_audit)\.log$

Note: This is working without additional configuration in version 4.1.

View solution in original post

Champion

Oreoshake, your reasoning is correct. The reason the SplunkLightForwarder/defaults are not working is because they are overridden by system/defaults. Thus, the desired configuration needs to be moved to a local/inputs.conf.

0 Karma

Champion

As mentioned, this could be a bug. It would be useful if you can post the version you are encountering this behavior in so we can pinpoint whether you are encountering a bug.

0 Karma

Champion

For clarification, this is applies to Lightweight Forwarder. The default expected behavior (LWF forwards its internal logs as default) is not working in version 4.0.x. The workaround is to create a monitor input for the Splunk logs directory and whitelist specific files in the $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *
_whitelist = (splunkd|metrics|license_audit)\.log$

Note: This is working without additional configuration in version 4.1.

View solution in original post