Getting Data In

How to drop events on a heavy forwarder?

vinaypradhan
Explorer

hi, 
I have a heavy forwarder configured this way

inputs.conf
[udp://514]
sourcetype = syslog
index = abc
connection_host = ip
disabled = 0
_TCP_ROUTING =MyRoute

So any server sending syslog on UDP 514 is hitting my heavy forwarder and getting sent to my indexers. 
I want to block events from a set of servers, as I don't want them to be indexed. They are consuming my licenses. 
I am trying to do this in props.conf and transforms.conf

props.conf
[host::XX.XX.XXX.XXX]
TRANSFORMS-abc = drop_events

and in my transforms.conf, I have this 

[drop_events]
REGEX =.
DEST_KEY = queue
FORMAT = nullQueue

But this is not dropping the events from this server. I can still events being indexed real time when i search. 

What am I doing wrong? I have a few more servers which I need to block this way. 

Any help is greatly appreciated. 


Labels (1)
0 Karma
1 Solution

vinaypradhan
Explorer

For syslog events, dropping events works in a different way. I was able to finally get it to work. Posting it here, so that it might help someone else who is facing the same kind of situation. 

Because of the way Splunk identifies and cooks a syslog, the syslog event ends up getting cooked using one of the configurations in the default props.conf called syslog-host
We will have to override that so that the events get dropped first before Splunk tries to cook the event. 

For example, if we want to drop all syslog events coming from server that have expr in their names, then this needs to be done this way

Ad this to prop.sconf

#syslog
[syslog]
TRANSFORMS = drop_events_syslog, syslog-host

And add drop_events_syslog in transforms.conf:

[drop_events_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(.*expr.*)\]?\s
DEST_KEY = queue
FORMAT = nullQueue

The most important thing here is that, in props.conf, we have to mention our drop stanza first and then also mention the syslog-host stanza. This will ensure syslog-host stanza gets handled after drop gets handled.

View solution in original post

vinaypradhan
Explorer

For syslog events, dropping events works in a different way. I was able to finally get it to work. Posting it here, so that it might help someone else who is facing the same kind of situation. 

Because of the way Splunk identifies and cooks a syslog, the syslog event ends up getting cooked using one of the configurations in the default props.conf called syslog-host
We will have to override that so that the events get dropped first before Splunk tries to cook the event. 

For example, if we want to drop all syslog events coming from server that have expr in their names, then this needs to be done this way

Ad this to prop.sconf

#syslog
[syslog]
TRANSFORMS = drop_events_syslog, syslog-host

And add drop_events_syslog in transforms.conf:

[drop_events_syslog]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(.*expr.*)\]?\s
DEST_KEY = queue
FORMAT = nullQueue

The most important thing here is that, in props.conf, we have to mention our drop stanza first and then also mention the syslog-host stanza. This will ensure syslog-host stanza gets handled after drop gets handled.

vinaypradhan
Explorer

I even tried changing [host::XX.XX.XXX.XXX] to [host::hostname] but still no luck

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...