Getting Data In

How to drop events on a heavy forwarder?

vinaypradhan
Explorer

hi, 
I have a heavy forwarder configured this way

inputs.conf
[udp://514]
sourcetype = syslog
index = abc
connection_host = ip
disabled = 0
_TCP_ROUTING =MyRoute

So any server sending syslog on UDP 514 is hitting my heavy forwarder and getting sent to my indexers. 
I want to block events from a set of servers, as I don't want them to be indexed. They are consuming my licenses. 
I am trying to do this in props.conf and transforms.conf

props.conf
[host::XX.XX.XXX.XXX]
TRANSFORMS-abc = drop_events

and in my transforms.conf, I have this 

[drop_events]
REGEX =.
DEST_KEY = queue
FORMAT = nullQueue

But this is not dropping the events from this server. I can still events being indexed real time when i search. 

What am I doing wrong? I have a few more servers which I need to block this way. 

Any help is greatly appreciated. 


Labels (1)
0 Karma
1 Solution

vinaypradhan
Explorer

For syslog events, dropping events works in a different way. I was able to finally get it to work. Posting it here, so that it might help someone else who is facing the same kind of situation. 

Because of the way Splunk identifies and cooks a syslog, the syslog event ends up getting cooked using one of the configurations in the default props.conf called syslog-host
We will have to override that so that the events get dropped first before Splunk tries to cook the event. 

For example, if we want to drop all syslog events coming from server that have expr in their names, then this needs to be done this way

Ad this to prop.sconf

#syslog
[syslog]
TRANSFORMS = drop_events_syslog, syslog-host

And add drop_events_syslog in transforms.conf:

[drop_events_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(.*expr.*)\]?\s
DEST_KEY = queue
FORMAT = nullQueue

The most important thing here is that, in props.conf, we have to mention our drop stanza first and then also mention the syslog-host stanza. This will ensure syslog-host stanza gets handled after drop gets handled.

View solution in original post

vinaypradhan
Explorer

For syslog events, dropping events works in a different way. I was able to finally get it to work. Posting it here, so that it might help someone else who is facing the same kind of situation. 

Because of the way Splunk identifies and cooks a syslog, the syslog event ends up getting cooked using one of the configurations in the default props.conf called syslog-host
We will have to override that so that the events get dropped first before Splunk tries to cook the event. 

For example, if we want to drop all syslog events coming from server that have expr in their names, then this needs to be done this way

Ad this to prop.sconf

#syslog
[syslog]
TRANSFORMS = drop_events_syslog, syslog-host

And add drop_events_syslog in transforms.conf:

[drop_events_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(.*expr.*)\]?\s
DEST_KEY = queue
FORMAT = nullQueue

The most important thing here is that, in props.conf, we have to mention our drop stanza first and then also mention the syslog-host stanza. This will ensure syslog-host stanza gets handled after drop gets handled.

vinaypradhan
Explorer

I even tried changing [host::XX.XX.XXX.XXX] to [host::hostname] but still no luck

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...