topic Re: How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder in Getting Data In

How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

kalianov — Tue, 29 Sep 2020 11:03:59 GMT

Hi.
My configuration is UF->HF->INDEXER.

Aim: configure DMC to monitor all instances of my deployment including Universal Forwarders (ver 6.1.4 or 6.2.0).
Problem is that I can't get splunkd.log and other internal logs from UniversalForwarders to my indexer(ver 6.4.1).
I have deployed a small app to my Universal Forwarders with such

inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.]
index = _internal
sourcetype = splunkd
_TCP_ROUTING = *
otputs.conf
[tcpout]
forwardedindex.0.whitelist = .
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false

But I still have no data on my indexer from that UF

On Universal Forwarders I have such $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = heavyforwarder:9997
[tcpout-server://heavyforwarder:9997]

All non internal logs have indexed good, but internal logs such as splunkd.log have not indexed.

Also I have some UFs that are sending data directly to indexer and I see all internal logs from them without my app. So I can monitor them and my heavy forvarder in DMC without problem, but I need all forwarders.

Need help

Re: How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

jtacy — Wed, 21 Sep 2016 17:17:27 GMT

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad

Re: How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

kalianov — Thu, 22 Sep 2016 09:24:27 GMT

Thanks a lot. I have did the next things on my HF:
- uninstalled my app, as you said
- copy from default outputs.conf some stanzas into system/local/outputs.conf :
[tcpout]
defaultGroup = myindexer:port

 maxQueueSize = auto
 forwardedindex.0.whitelist = .*
 forwardedindex.1.blacklist = _.*
 forwardedindex.2.whitelist = (_internal)

disable that lines in default outputs.conf
restart heavy Forwarder

It is works

I hope that my license will not be down.

Re: How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

saurabh_tek — Mon, 10 Apr 2017 10:36:40 GMT

License is not counted against splunk's own 'internal' logs.

Re: How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

hsesterhenn_spl — Mon, 13 Aug 2018 06:49:57 GMT

Hi,

just found this discussion...

I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-).

The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do.

You found the black/whitelisting.

Much easier would be:

forwardedindex.filter.disable = true

which if off (=false) by default.

This is needed on the intermediate HF in your case.

Hope to help others for future reference.

Happy splunking,

Holger