Getting Data In

How to fix my universal forwarder configurations so that Splunk only forwards the data I want to monitor to a third-party system?

anton085
Path Finder

I am trying to forward to a third-party system from a Universal forwarder. I have tried two approaches. In both cases I am receiving a lot of unnecessary data on the third-party end. It looks like Splunk is not only forwarding the file that I am monitoring but also internal logs as well. What can I do to fix this? I am attaching conf files for both:

Approach 1: use props, transforms, and outputs

props.conf
[source::/home/abc/splunk-test/test.txt]
TRANSFORMS-routing=monitoring

transforms.conf
[monitoring]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

Approach 2: use inputs.conf and outputs.conf (I deleted everything from props and transforms)
inputs.conf
[default]
host=abc

[monitor:///home/abc/splunk-test/test.txt]
_TCP_ROUTING=monitoring_tcp

outputs.conf
[tcpout]
defaultGroup=group_a

[tcpout:group_a]
disabled=true

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514

0 Karma
1 Solution

anton085
Path Finder

I have figured out two ways to block internal logs from being forwarded:

  1. inputs.conf
    use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
    disabled=true
    [monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
    disabled=true

  2. outputs.conf
    use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
    So, here, the whitelist would win

    forwardedindex.0.whitelist
    forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

View solution in original post

anton085
Path Finder

I have figured out two ways to block internal logs from being forwarded:

  1. inputs.conf
    use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default

    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk]
    disabled=true
    [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
    disabled=true
    [monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
    disabled=true

  2. outputs.conf
    use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
    So, here, the whitelist would win

    forwardedindex.0.whitelist
    forwardedindex.0.blacklist

And here, the blacklist would win

forwardedindex.0.whitelist
forwardedindex.1.blacklist

From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.

[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)

kutzi
Path Finder

I tried method 1, but it doesn't seem to work.
I put

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled=true

into $SPLUNK_HOMe/etc/system/local/inputs.conf

Did you put your inputs.conf somewhere else?

0 Karma

sbbadri
Motivator

[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
forwardedindex.0.blacklist = (_internal|_audit)

please go through below link for more details,

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

0 Karma

anton085
Path Finder

I have tried this and it didn't work. Moreover, the link said to add blacklist under a global tcp stanza only, I tried that and that didn't work either.

0 Karma

anton085
Path Finder

actually added the following lines in outputs.conf but to no avail:

[tcpout]
forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection)
forwardedindex.2.blacklist = (_internal|_audit|_telemetry|_introspection)

0 Karma
Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...