Easiest way to exclude ingestion of events for a specific IP address from a SourceType?

[neiowe]

I am looking to not ingest events from a specific IP address. I have an IP address that once a week generates a LOT of firewall traffic and which is causing me to exceed my license. What's the easiest/best way to exclude these events. The firewall logs are being sent to a SYSLOG server that is running a Universal Forwarder that sends all logs to my Indexer. I am assuming I will need to use the PROPS/TRANSFORM files to send these to the NULL Queue, but not sure of the proper stanza's to accomplish this. Anyone done this and can provide a sample or suggest a better way to exclude these events?

[493669]

@neiowe can you try below:
in transforms.conf:

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

in props.conf:

[pan:traffic]
TRANSFORMS-null= setnull

[neiowe]

[micahkemp]

Does your input set sourcetype=pan:traffic, or does it set sourcetype=pan:log, which is then changed to pan:traffic via a TRANSFORM in the Palo Alto TA?

I'm guessing the latter, which means that the transforms you define for the pan:traffic sourcetype will not be run.

When an event traverses is indexed it will do so for the sourcetype defined when it enters the indexing queue, and changing the sourcetype via a transform will not result in the new sourcetype's transforms being run.

[neiowe]

Bingo! That was the problem. I checked my inputs.conf and the sourcetype was being set to palo:log. I change the sourcetype to palo:log in the transforms and it's working perfect now. Thank you very much!

[mayurr98 · Accepted solution]

Hey
This is done by defining a regex to match the necessary event(s) and send them to nullqueue

Here is a basic example that will drop everything the events that you do not want.
Let suppose you have ip 192.168.10.11 in the event which generate all these firewall logs and you want exclude these events

Then put:
in transforms.conf

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

And in props.conf

 [your_sourcetype]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-null = setnull

Let me know if this helps!

[elvis5]

Good day .

Let suppose ihave ip 172.16.1.50, 192.168.1.50 and 192.168.2.0/24 in the event which generate all these firewall logs and i want exclude these events. What syntaxis in REGEX

[PickleRick]

1. Please don't hijack old threads (especially ancient ones like this one)

2. You can't (reasonably) match CIDR ranges using regex. With /24 you can simply cut at last octet but with - for example /17... it's gonna be a pain in the posterior.

[himanshu1]

@mayurr98 Hi ,

How to write regex for ip range and can we use wildcard here ?

[setnull]
REGEX = 192\.168\.10\.11
DEST_KEY = queue
FORMAT = nullQueue

 Thanks in advance 🙂

[mayurr98]

Hey

I think according to the documentation its TRANSFORMS-null = setnull
And in your screenshot you had written TRANSFORMS-set = setnull

So there was some syntax error .Change this and restart the instance. It should work

[neiowe]

@mayurr98, thank you very much for all your assistance! You original solution worked perfect, I just had the wrong sourcetype in the transforms as pointed out by @micahkemp below. Thanks to you both for helping be resolve this.

[mayurr98]

Hey @neiowe

Glad to help you!! Pls upvote as well

[micahkemp]

The class name of the transforms setting should not matter.

[mayurr98]

Try
index=your_index “10.3.30.35”

Do a real time search and see if you getting events

[neiowe]

Yes, I am still get real-time events. Just a side questions: Does a change to transform.conf require a restart of Splunk? I have been doing it but just want to confirm that it is necessary.

[mayurr98]

Yes restart is must.It is given in doc as well

Refer this doc to exactly same as mentioned in this doc
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad

Look for Discard specific events and keep the rest in Filter event data and send to queues.

Keep all the naming convention same as given in doc and try it must work as i have tried and worked 100%

I hope this solves your problem !

[neiowe]

This doesn't appear to be working for me. I tried for just the one IP address first. Here's what I added to the transforms/props on my indexers. I also restarted Splunk on the indexers. What am I missing?

In transforms.conf

[palo_setnull]
REGEX = 10.3.30.35
DEST_KEY = queue
FORMAT = nullQueue

In props.conf

[pan:traffic]
TRANSFORMS-set = palo_setnull

[mayurr98]

Could you please share the sample event of this ip? Cause regex should match the sample event

Also , could you please keep just setnull instead of palo_setnull and restart the instance?

Also palo:traffic is the name of your sourcetype?

[neiowe]

Jan 5 15:11:18 ISG-5020-FW1 1,2018/01/05 15:11:17,007701002549,TRAFFIC,drop,1,2018/01/05 15:11:17,10.3.30.35,172.31.155.128,0.0.0.0,0.0.0.0,Rule 43,aca\sa_qualsys,,not-applicable,vsys1,inside,DMZ31,ethernet1/2,,ISG-LOG,2018/01/05 15:11:17,0,1,7712,12374,0,0,0x4000,udp,deny,60,60,0,1,2018/01/05 15:11:18,0,any,0,1892346490,0x0,10.0.0.0-10.255.255.255,172.16.0.0-172.31.255.255,0,1,0,policy-deny,12,0,0,0,,ISG-5020-FW1,from-policy

I already have a setnull but I can change to setnull2 if you think it will make a difference.

[mayurr98]

Put 10\.3\.30\.35
Also one more thing, are you looking historical events or real time events cause this exercise will exclude future events and not the historical events so look for real time events for this ip.

[neiowe]

I actually do have 10.3.30.35 but for some reason it didn't show up above when I copied the editor. I am looking at real-time events.