UPDATE: This appears to be a bug specifically related to 4.0.10. The following is a work around in system/local/inputs.conf
[monitor:///home/ops/splunk/var/log/splunk]
disabled=true
It looks like the entry in system/default is clobbering the more granular entries in SplunkLightForwarder/default. The output of splunk list monitor seems to be broken as well
I'd like to index the splunkd.log for remote troubleshooting but I can't get my light forwarders to forward the log. Here's the btool output (with unrelated stuff removed):
[root@neil bin]# ./splunk cmd btool --debug inputs list
system [default]
system _rcvbuf = 1572864
system host = myforwarder
system index = default
system [monitor:///home/ops/splunk/var/log/splunk]
system _rcvbuf = 1572864
system host = myforwarder
system index = _internal
SplunkLigh [monitor:///home/ops/splunk/var/log/splunk/splunkd.log]
SplunkLigh _TCP_ROUTING = *
system _rcvbuf = 1572864
system host = myforwarder
maint index = _internal
maint sourcetype = splunkd
atti-linux [monitor:///var/log]
system _rcvbuf = 1572864
atti-linux _whitelist = (auth$|cron$|kern$|lpr$|maillog$|user$|local$)
system host = myforwarder
system index = default
system [splunktcp]
system _rcvbuf = 1572864
system host = myforwarder
system index = default
system route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue
I tried changing the index, no dice. From what I can tell, the /var/log/splunk entry clobbers the more specific one which says to include the entire var/log/splunk directory. When I run "splunk list monitor" it prints out the entire var/log/splunk dir except splunkd.log!
[root@neil-search etc]# ../bin/splunk list monitor
Monitored Directories:
$SPLUNK_HOME/var/log/splunk
/home/ops/splunk/var/log/splunk/audit.log
/home/ops/splunk/var/log/splunk/btool.log
/home/ops/splunk/var/log/splunk/metrics.log.1
/home/ops/splunk/var/log/splunk/metrics.log.2
/home/ops/splunk/var/log/splunk/metrics.log.3
/home/ops/splunk/var/log/splunk/metrics.log.4
/home/ops/splunk/var/log/splunk/migration.log.2010-03-23.00-14-09
/home/ops/splunk/var/log/splunk/migration.log.2010-03-23.17-02-12
/home/ops/splunk/var/log/splunk/migration.log.2010-03-26.18-58-16
/home/ops/splunk/var/log/splunk/searchhistory.log
/home/ops/splunk/var/log/splunk/splunkd_access.log
/home/ops/splunk/var/log/splunk/splunkd_stderr.log
/home/ops/splunk/var/log/splunk/splunkd_stdout.log
/home/ops/splunk/var/log/splunk/splunklogger.log
For clarification, this is applies to Lightweight Forwarder. The default expected behavior (LWF forwards its internal logs as default) is not working in version 4.0.x. The workaround is to create a monitor input for the Splunk logs directory and whitelist specific files in the $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *
_whitelist = (splunkd|metrics|license_audit)\.log$
Note: This is working without additional configuration in version 4.1.
Oreoshake, your reasoning is correct. The reason the SplunkLightForwarder/defaults are not working is because they are overridden by system/defaults. Thus, the desired configuration needs to be moved to a local/inputs.conf.
As mentioned, this could be a bug. It would be useful if you can post the version you are encountering this behavior in so we can pinpoint whether you are encountering a bug.
For clarification, this is applies to Lightweight Forwarder. The default expected behavior (LWF forwards its internal logs as default) is not working in version 4.0.x. The workaround is to create a monitor input for the Splunk logs directory and whitelist specific files in the $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *
_whitelist = (splunkd|metrics|license_audit)\.log$
Note: This is working without additional configuration in version 4.1.