Hi.
My configuration is UF->HF->INDEXER.
Aim: configure DMC to monitor all instances of my deployment including Universal Forwarders (ver 6.1.4 or 6.2.0).
Problem is that I can't get splunkd.log and other internal logs from UniversalForwarders to my indexer(ver 6.4.1).
I have deployed a small app to my Universal Forwarders with such
inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.]
index = _internal
sourcetype = splunkd
_TCP_ROUTING = *
otputs.conf
[tcpout]
forwardedindex.0.whitelist = .
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false
But I still have no data on my indexer from that UF
On Universal Forwarders I have such $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = heavyforwarder:9997
[tcpout-server://heavyforwarder:9997]
All non internal logs have indexed good, but internal logs such as splunkd.log have not indexed.
Also I have some UFs that are sending data directly to indexer and I see all internal logs from them without my app. So I can monitor them and my heavy forvarder in DMC without problem, but I need all forwarders.
Need help
It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.
If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.
For what it's worth, the documentation suggests this outputs.conf to forward all indexes:
#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad
Hi,
just found this discussion...
I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-).
The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do.
You found the black/whitelisting.
Much easier would be:
forwardedindex.filter.disable = true
which if off (=false) by default.
This is needed on the intermediate HF in your case.
Hope to help others for future reference.
Happy splunking,
Holger
Thanks a lot. I have did the next things on my HF:
- uninstalled my app, as you said
- copy from default outputs.conf some stanzas into system/local/outputs.conf :
[tcpout]
defaultGroup = myindexer:port
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_internal)
It is works
I hope that my license will not be down.
License is not counted against splunk's own 'internal' logs.
It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.
If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.
For what it's worth, the documentation suggests this outputs.conf to forward all indexes:
#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad