Getting Data In

How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085
Path Finder

I am trying to forward to a third-party system from a Universal forwarder. I have tried two approaches. In both cases I am receiving a lot of unnecessary data on the third-party end. It looks like Splunk is not only forwarding the file that I am monitoring but also internal logs as well. What can I do to fix this? I am attaching conf files for both:

Approach 1: use props, transforms, and outputs

props.conf
[source::/home/abc/splunk-test/test.txt]
TRANSFORMS-routing=monitoring

transforms.conf
[monitoring]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

Approach 2: use inputs.conf and outputs.conf (I deleted everything from props and transforms)
inputs.conf
[default]
host=abc

[monitor:///home/abc/splunk-test/test.txt]
_TCP_ROUTING=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

0 Karma
1 Solution

anton085
Path Finder

I have figured out two ways to block internal logs from being forwarded:

  1. inputs.conf
    use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
    disabled=true
    [monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
    disabled=true

  2. outputs.conf
    use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
    So, here, the whitelist would win

    forwardedindex.0.whitelist
    forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

View solution in original post

anton085
Path Finder

I have figured out two ways to block internal logs from being forwarded:

  1. inputs.conf
    use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
    disabled=true
    [monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
    disabled=true

  2. outputs.conf
    use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
    So, here, the whitelist would win

    forwardedindex.0.whitelist
    forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

kutzi
Path Finder

I tried method 1, but it doesn't seem to work.
I put

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled=true

into $SPLUNK_HOMe/etc/system/local/inputs.conf

Did you put your inputs.conf somewhere else?

0 Karma

sbbadri
Motivator

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
forwardedindex.0.blacklist = (_internal|_audit)

please go through below link for more details,

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

0 Karma

anton085
Path Finder

I have tried this and it didn't work. Moreover, the link said to add blacklist under a global tcp stanza only, I tried that and that didn't work either.

0 Karma

anton085
Path Finder

actually added the following lines in outputs.conf but to no avail:

[tcpout]
forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection)
forwardedindex.2.blacklist = (_internal|_audit|_telemetry|_introspection)

0 Karma

Kasee
New Member

The default setting for the inputs.conf for the UF is a wildcard.  Change this to the default group in the local inputs.conf to override the setting.

Pulled from the default inputs.conf for the UF:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log] 

_TCP_ROUTING = *

index = _internal 

  

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log] 

_TCP_ROUTING = *

index = _internal 

Add it in the local inputs.conf  and change it to what ever the default group is on the outputs.conf, based on what was put in the thread appears to be group_a:

 

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log] 

_TCP_ROUTING = group_a

index = _internal 

  

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log] 

_TCP_ROUTING = group_a

index = _internal 

 

This should eliminate the _internal logs from being forwarded to the 3rd Party system.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...