How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

[anton085]

I am trying to forward to a third-party system from a Universal forwarder. I have tried two approaches. In both cases I am receiving a lot of unnecessary data on the third-party end. It looks like Splunk is not only forwarding the file that I am monitoring but also internal logs as well. What can I do to fix this? I am attaching conf files for both:

Approach 1: use props, transforms, and outputs

props.conf
[source::/home/abc/splunk-test/test.txt]
TRANSFORMS-routing=monitoring

transforms.conf
[monitoring]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

Approach 2: use inputs.conf and outputs.conf (I deleted everything from props and transforms)
inputs.conf
[default]
host=abc

[monitor:///home/abc/splunk-test/test.txt]
_TCP_ROUTING=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

[anton085 · Accepted solution]

I have figured out two ways to block internal logs from being forwarded:

1. inputs.conf
   use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

   [monitor://$SPLUNK_HOME/var/log/splunk]
   disabled=true
   [monitor://$SPLUNK_HOME/var/log/splunk]
   disabled=true
   [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
   disabled=true
   [monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
   disabled=true

2. outputs.conf
   use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
   So, here, the whitelist would win

   forwardedindex.0.whitelist
   forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

[kutzi]

I tried method 1, but it doesn't seem to work.
I put

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled=true

into $SPLUNK_HOMe/etc/system/local/inputs.conf

Did you put your inputs.conf somewhere else?

[sbbadri]

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
forwardedindex.0.blacklist = (_internal|_audit)

please go through below link for more details,

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

[anton085]

I have tried this and it didn't work. Moreover, the link said to add blacklist under a global tcp stanza only, I tried that and that didn't work either.

[anton085]

actually added the following lines in outputs.conf but to no avail:

[tcpout]
forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection)
forwardedindex.2.blacklist = (_internal|_audit|_telemetry|_introspection)

[Kasee]

The default setting for the inputs.conf for the UF is a wildcard.  Change this to the default group in the local inputs.conf to override the setting.

Pulled from the default inputs.conf for the UF:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log] 

_TCP_ROUTING = *

index = _internal 

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log] 

_TCP_ROUTING = *

index = _internal 

Add it in the local inputs.conf  and change it to what ever the default group is on the outputs.conf, based on what was put in the thread appears to be group_a:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log] 

_TCP_ROUTING = group_a

index = _internal 

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log] 

_TCP_ROUTING = group_a

index = _internal 

This should eliminate the _internal logs from being forwarded to the 3rd Party system.