How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

[kalianov]

Hi.
My configuration is UF->HF->INDEXER.

Aim: configure DMC to monitor all instances of my deployment including Universal Forwarders (ver 6.1.4 or 6.2.0).
Problem is that I can't get splunkd.log and other internal logs from UniversalForwarders to my indexer(ver 6.4.1).
I have deployed a small app to my Universal Forwarders with such

inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.]
index = _internal
sourcetype = splunkd
_TCP_ROUTING = *
otputs.conf
[tcpout]
forwardedindex.0.whitelist = .
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false

But I still have no data on my indexer from that UF

On Universal Forwarders I have such $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = heavyforwarder:9997
[tcpout-server://heavyforwarder:9997]

All non internal logs have indexed good, but internal logs such as splunkd.log have not indexed.

Also I have some UFs that are sending data directly to indexer and I see all internal logs from them without my app. So I can monitor them and my heavy forvarder in DMC without problem, but I need all forwarders.

Need help

[hsesterhenn_spl]

Hi,

just found this discussion...

I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-).

The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do.

You found the black/whitelisting.

Much easier would be:

forwardedindex.filter.disable = true

which if off (=false) by default.

This is needed on the intermediate HF in your case.

Hope to help others for future reference.

Happy splunking,

Holger

[kalianov]

Thanks a lot. I have did the next things on my HF:
- uninstalled my app, as you said
- copy from default outputs.conf some stanzas into system/local/outputs.conf :
[tcpout]
defaultGroup = myindexer:port

 maxQueueSize = auto
 forwardedindex.0.whitelist = .*
 forwardedindex.1.blacklist = _.*
 forwardedindex.2.whitelist = (_internal)

• disable that lines in default outputs.conf
• restart heavy Forwarder

It is works

I hope that my license will not be down.

[saurabh_tek]

License is not counted against splunk's own 'internal' logs.

[jtacy · Accepted solution]

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad