Getting Data In

How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

kalianov
Path Finder

Hi.
My configuration is UF->HF->INDEXER.

Aim: configure DMC to monitor all instances of my deployment including Universal Forwarders (ver 6.1.4 or 6.2.0).
Problem is that I can't get splunkd.log and other internal logs from UniversalForwarders to my indexer(ver 6.4.1).
I have deployed a small app to my Universal Forwarders with such

inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.]
index = _internal
sourcetype = splunkd
_TCP_ROUTING = *
otputs.conf
[tcpout]
forwardedindex.0.whitelist = .

forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false

But I still have no data on my indexer from that UF

On Universal Forwarders I have such $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = heavyforwarder:9997
[tcpout-server://heavyforwarder:9997]

All non internal logs have indexed good, but internal logs such as splunkd.log have not indexed.

Also I have some UFs that are sending data directly to indexer and I see all internal logs from them without my app. So I can monitor them and my heavy forvarder in DMC without problem, but I need all forwarders.

Need help

0 Karma
1 Solution

jtacy
Builder

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad

View solution in original post

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi,

just found this discussion...

I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-).

The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do.

You found the black/whitelisting.

Much easier would be:

forwardedindex.filter.disable = true

which if off (=false) by default.

This is needed on the intermediate HF in your case.

Hope to help others for future reference.

Happy splunking,

Holger

kalianov
Path Finder

Thanks a lot. I have did the next things on my HF:
- uninstalled my app, as you said
- copy from default outputs.conf some stanzas into system/local/outputs.conf :
[tcpout]
defaultGroup = myindexer:port

 maxQueueSize = auto
 forwardedindex.0.whitelist = .*
 forwardedindex.1.blacklist = _.*
 forwardedindex.2.whitelist = (_internal)
  • disable that lines in default outputs.conf
  • restart heavy Forwarder

It is works

I hope that my license will not be down.

saurabh_tek
Communicator

License is not counted against splunk's own 'internal' logs.

0 Karma

jtacy
Builder

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...