Getting Data In

[udp://:portnumber] Event Blacklist- How do I prevent unwanted data from being indexed?

LinghGroove
Explorer

Hello,

I am currently receiving firewall data on my heavy forwarder on a specific port number. On the HF there is an simple inputs.conf with

[udp://:portnumber]
sourcetype=fgt_log
index=fw_data

and an outputs.conf that sends all to the indexers. The problem is that i am receiving a lot of garbage traffic (like DNS traffic to 8.8.8.8 or 8.8.4.4). I don't want to index this data. I don't have access to the firewall so i can't just stop it there. 

I thought that a blacklist would stop the events from coming. (I tried a simple blacklist like the ones used under the [monitor] stanza. Something like this: blacklist=(dstip=8\.8\.8\.8|dstip=8\.8\.4\.4|service="DNS"))

Unfortunately it didn't work... I made some research but i only found the "acceptFrom" that in this situation i don't think it's useful. Came across this post but wasn't useful https://community.splunk.com/t5/Getting-Data-In/Blacklist-a-host-hosts-is-sending-logs-to-Splunk-via...

Any tips? 

Labels (4)
0 Karma
1 Solution

LinghGroove
Explorer

@maciep Ok after some test i figured out how to make it work. 

PROPS

[fgt_log]                          #here you have to put the sourcetype of your data, not the source like the documentation tells you
TRANSFORMS-null = transnull        #reference to the transform stanza for the nullqueue   

here the syntax is TRANSFORMS-<class> and for what i have understood the <class>name can be anything you want (except some special characters, if I'm not mistaken), same for the stanza name in the transform.

TRANSFORMS

[transnull]
REGEX = (dstip=(?:8\.8\.8\.8|8\.8\.4\.4)[^D]+D(?:NS)?(?:[^D\n]+DNS)?)
DEST_KEY = queue
FORMAT = nullQueue

 Here i had a bit of trouble with the regex syntax but in the end i made it work. DEST_KEY like the documentation says, same for FORMAT. The name of the stanza is the same called in the props.conf.

INPUTS

[udp://:myport]
sourcetype = fgt_log
source = ***
index = ***

 With this it works all perfectly. Thaks again for the support @maciep 🙂

View solution in original post

maciep
Champion

There used to be a really good .conf presentation on regex, but I guess Splunk decided to remove all of the old .conf content from their site....such a shame.  Glad you were able to make it work!

maciep
Champion

I am not sure about the udp input stanza and whether you can blacklist there or not.  But a common approach is to use props/transforms to filter matching events to the "nullQueue" so that they are not ingested.  That is something that can be done on the heavy forwarder.

You can find details in the docs about it but also probably quite a few posts out here detailing the steps...
Route and filter data - Splunk Documentation

LinghGroove
Explorer

Hello @maciep thanks for your reply,

I have tried the solution you are suggesting... unfortunatly i am not able to make it work... I just now tried the solution from https://community.splunk.com/t5/Getting-Data-In/How-could-i-filter-network-firewall-data-using-a-fil...

The problem presented in this question is nearly identical. I tried to apply it just as they answered but i doesn't work. Maybe i'm setting something wrong or maybe i am putting a wrong regex. If i'll make it work i will write you back here.

Thanks again

 

0 Karma

LinghGroove
Explorer

@maciep Ok after some test i figured out how to make it work. 

PROPS

[fgt_log]                          #here you have to put the sourcetype of your data, not the source like the documentation tells you
TRANSFORMS-null = transnull        #reference to the transform stanza for the nullqueue   

here the syntax is TRANSFORMS-<class> and for what i have understood the <class>name can be anything you want (except some special characters, if I'm not mistaken), same for the stanza name in the transform.

TRANSFORMS

[transnull]
REGEX = (dstip=(?:8\.8\.8\.8|8\.8\.4\.4)[^D]+D(?:NS)?(?:[^D\n]+DNS)?)
DEST_KEY = queue
FORMAT = nullQueue

 Here i had a bit of trouble with the regex syntax but in the end i made it work. DEST_KEY like the documentation says, same for FORMAT. The name of the stanza is the same called in the props.conf.

INPUTS

[udp://:myport]
sourcetype = fgt_log
source = ***
index = ***

 With this it works all perfectly. Thaks again for the support @maciep 🙂

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...