topic Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system? in Getting Data In

How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085 — Tue, 29 Sep 2020 15:07:18 GMT

I am trying to forward to a third-party system from a Universal forwarder. I have tried two approaches. In both cases I am receiving a lot of unnecessary data on the third-party end. It looks like Splunk is not only forwarding the file that I am monitoring but also internal logs as well. What can I do to fix this? I am attaching conf files for both:

Approach 1: use props, transforms, and outputs

props.conf
[source::/home/abc/splunk-test/test.txt]
TRANSFORMS-routing=monitoring

transforms.conf
[monitoring]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

Approach 2: use inputs.conf and outputs.conf (I deleted everything from props and transforms)
inputs.conf
[default]
host=abc

[monitor:///home/abc/splunk-test/test.txt]
_TCP_ROUTING=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

sbbadri — Tue, 29 Sep 2020 15:10:29 GMT

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
forwardedindex.0.blacklist = (_internal|_audit)

please go through below link for more details,

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085 — Mon, 31 Jul 2017 21:05:14 GMT

I have tried this and it didn't work. Moreover, the link said to add blacklist under a global tcp stanza only, I tried that and that didn't work either.

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085 — Tue, 29 Sep 2020 15:07:20 GMT

actually added the following lines in outputs.conf but to no avail:

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085 — Tue, 29 Sep 2020 15:08:47 GMT

I have figured out two ways to block internal logs from being forwarded:

inputs.conf
use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled=true
[monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
disabled=true
outputs.conf
use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
So, here, the whitelist would win

forwardedindex.0.whitelist
forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

kutzi — Tue, 23 Apr 2019 12:59:51 GMT

I tried method 1, but it doesn't seem to work.
I put

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled=true

into $SPLUNK_HOMe/etc/system/local/inputs.conf

Did you put your inputs.conf somewhere else?

Re: How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-

Kasee — Mon, 10 Jul 2023 16:05:43 GMT

The default setting for the inputs.conf for the UF is a wildcard. Change this to the default group in the local inputs.conf to override the setting.

Pulled from the default inputs.conf for the UF:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]

_TCP_ROUTING = *

index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]

_TCP_ROUTING = *

index = _internal

Add it in the local inputs.conf and change it to what ever the default group is on the outputs.conf, based on what was put in the thread appears to be group_a:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]

_TCP_ROUTING = group_a

index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]

_TCP_ROUTING = group_a

index = _internal

This should eliminate the _internal logs from being forwarded to the 3rd Party system.