Getting Data In

How to forward internal logs (splunkd.log) from UniversalForwarders to indexer via heavy forwarder

kalianov
Path Finder

Hi.
My configuration is UF->HF->INDEXER.

Aim: configure DMC to monitor all instances of my deployment including Universal Forwarders (ver 6.1.4 or 6.2.0).
Problem is that I can't get splunkd.log and other internal logs from UniversalForwarders to my indexer(ver 6.4.1).
I have deployed a small app to my Universal Forwarders with such

inputs.conf:
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.]
index = _internal
sourcetype = splunkd
_TCP_ROUTING = *
otputs.conf
[tcpout]
forwardedindex.0.whitelist = .

forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false

But I still have no data on my indexer from that UF

On Universal Forwarders I have such $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = heavyforwarder:9997
[tcpout-server://heavyforwarder:9997]

All non internal logs have indexed good, but internal logs such as splunkd.log have not indexed.

Also I have some UFs that are sending data directly to indexer and I see all internal logs from them without my app. So I can monitor them and my heavy forvarder in DMC without problem, but I need all forwarders.

Need help

0 Karma
1 Solution

jtacy
Builder

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad

View solution in original post

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi,

just found this discussion...

I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-).

The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do.

You found the black/whitelisting.

Much easier would be:

forwardedindex.filter.disable = true

which if off (=false) by default.

This is needed on the intermediate HF in your case.

Hope to help others for future reference.

Happy splunking,

Holger

kalianov
Path Finder

Thanks a lot. I have did the next things on my HF:
- uninstalled my app, as you said
- copy from default outputs.conf some stanzas into system/local/outputs.conf :
[tcpout]
defaultGroup = myindexer:port

 maxQueueSize = auto
 forwardedindex.0.whitelist = .*
 forwardedindex.1.blacklist = _.*
 forwardedindex.2.whitelist = (_internal)
  • disable that lines in default outputs.conf
  • restart heavy Forwarder

It is works

I hope that my license will not be down.

saurabh_tek
Communicator

License is not counted against splunk's own 'internal' logs.

0 Karma

jtacy
Builder

It looks like UF 6.1.4 and 6.2.0 will forward splunkd.log to all tcpout stanzas by default. I suspect that you don't need this custom app on your UF and that your HF is dropping your _internal events. If you've enabled the SplunkForwarder app on your HF, at least on 6.4.0 it contains an outputs.conf that will filter out _internal events.

If you remove your app from the UFs then deploy an app on the HF to allow forwarding of the _internal index (you just need an outputs.conf, I think you'll get what you need.

For what it's worth, the documentation suggests this outputs.conf to forward all indexes:

#Forward everything
[tcpout]
forwardedindex.0.whitelist = .*
# disable these
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

Source: http://docs.splunk.com/Documentation/Splunk/6.4.3/Forwarding/Routeandfilterdatad

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...