I am trying to forward to a third-party system from a Universal forwarder. I have tried two approaches. In both cases I am receiving a lot of unnecessary data on the third-party end. It looks like Splunk is not only forwarding the file that I am monitoring but also internal logs as well. What can I do to fix this? I am attaching conf files for both:
Approach 1: use props, transforms, and outputs
props.conf
[source::/home/abc/splunk-test/test.txt]
TRANSFORMS-routing=monitoring
transforms.conf
[monitoring]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=monitoring_tcp
outputs.conf
[tcpout]
defaultGroup=group_a
[tcpout:group_a]
disabled=true
[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
Approach 2: use inputs.conf and outputs.conf (I deleted everything from props and transforms)
inputs.conf
[default]
host=abc
[monitor:///home/abc/splunk-test/test.txt]
_TCP_ROUTING=monitoring_tcp
outputs.conf
[tcpout]
defaultGroup=group_a
[tcpout:group_a]
disabled=true
[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
I have figured out two ways to block internal logs from being forwarded:
inputs.conf
use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled=true
[monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
disabled=true
outputs.conf
use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
So, here, the whitelist would win
forwardedindex.0.whitelist
forwardedindex.0.blacklist
And here, the blacklist would win
forwardedindex.0.whitelist
forwardedindex.1.blacklist
From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.
[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)
I have figured out two ways to block internal logs from being forwarded:
inputs.conf
use disabled=true for inputs that have to be blocked. For me, I blocked these ones, which had _TCP_ROUTING=* set in the default inputs.conf files inside $SPLUNK_HOME/etc/system/default and $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled=true
[monitor://$SPLUNK_HOM/var/log/splunk/metrics.log]
disabled=true
outputs.conf
use the blacklist. According to the documentation, a whitelist can override a blacklist when both have the same number, and the filtering order is based on increasing number. Also, the filtering will only work under the [tcpout] stanza.
So, here, the whitelist would win
forwardedindex.0.whitelist
forwardedindex.0.blacklist
And here, the blacklist would win
forwardedindex.0.whitelist
forwardedindex.1.blacklist
From the default conf files, I figured out that there are 3 lists going from 0 to 2. So I added the following snippet in $SPLUNK_HOME/etc/system/local/outputs.conf, and it worked.
[tcpout]
forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection)
I tried method 1, but it doesn't seem to work.
I put
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled=true
into $SPLUNK_HOMe/etc/system/local/inputs.conf
Did you put your inputs.conf somewhere else?
[tcpout:monitoring_tcp]
sendCookedData=false
server=x.x.x.x:514
forwardedindex.0.blacklist = (_internal|_audit)
please go through below link for more details,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
I have tried this and it didn't work. Moreover, the link said to add blacklist under a global tcp stanza only, I tried that and that didn't work either.
actually added the following lines in outputs.conf but to no avail:
[tcpout]
forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection)
forwardedindex.2.blacklist = (_internal|_audit|_telemetry|_introspection)