Splunk Search

Easiest way to exclude ingestion of events for a specific IP address from a SourceType?

neiowe
Path Finder

I am looking to not ingest events from a specific IP address. I have an IP address that once a week generates a LOT of firewall traffic and which is causing me to exceed my license. What's the easiest/best way to exclude these events. The firewall logs are being sent to a SYSLOG server that is running a Universal Forwarder that sends all logs to my Indexer. I am assuming I will need to use the PROPS/TRANSFORM files to send these to the NULL Queue, but not sure of the proper stanza's to accomplish this. Anyone done this and can provide a sample or suggest a better way to exclude these events?

Labels (1)
0 Karma
1 Solution

mayurr98
Super Champion

Hey
This is done by defining a regex to match the necessary event(s) and send them to nullqueue

Here is a basic example that will drop everything the events that you do not want.
Let suppose you have ip 192.168.10.11 in the event which generate all these firewall logs and you want exclude these events

Then put:
in transforms.conf

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

And in props.conf

 [your_sourcetype]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-null = setnull

Let me know if this helps!

View solution in original post

493669
Super Champion

@neiowe can you try below:
in transforms.conf:

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

in props.conf:

[pan:traffic]
TRANSFORMS-null= setnull

neiowe
Path Finder

alt text

alt text

0 Karma

micahkemp
Champion

Does your input set sourcetype=pan:traffic, or does it set sourcetype=pan:log, which is then changed to pan:traffic via a TRANSFORM in the Palo Alto TA?

I'm guessing the latter, which means that the transforms you define for the pan:traffic sourcetype will not be run.

When an event traverses is indexed it will do so for the sourcetype defined when it enters the indexing queue, and changing the sourcetype via a transform will not result in the new sourcetype's transforms being run.

neiowe
Path Finder

Bingo! That was the problem. I checked my inputs.conf and the sourcetype was being set to palo:log. I change the sourcetype to palo:log in the transforms and it's working perfect now. Thank you very much!

0 Karma

mayurr98
Super Champion

Hey
This is done by defining a regex to match the necessary event(s) and send them to nullqueue

Here is a basic example that will drop everything the events that you do not want.
Let suppose you have ip 192.168.10.11 in the event which generate all these firewall logs and you want exclude these events

Then put:
in transforms.conf

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

And in props.conf

 [your_sourcetype]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-null = setnull

Let me know if this helps!

elvis5
Loves-to-Learn Lots

Good day .

Let suppose ihave ip 172.16.1.50, 192.168.1.50 and 192.168.2.0/24 in the event which generate all these firewall logs and i want exclude these events. What syntaxis in REGEX

 

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Please don't hijack old threads (especially ancient ones like this one)

2. You can't (reasonably) match CIDR ranges using regex. With /24 you can simply cut at last octet but with - for example /17... it's gonna be a pain in the posterior.

0 Karma

himanshu1
Loves-to-Learn Lots

@mayurr98 Hi ,

How to write regex for ip range and can we use wildcard here ?

[setnull]
REGEX = 192\.168\.10\.11
DEST_KEY = queue
FORMAT = nullQueue

 Thanks in advance 🙂

 

0 Karma

mayurr98
Super Champion

Hey

I think according to the documentation its TRANSFORMS-null = setnull
And in your screenshot you had written TRANSFORMS-set = setnull

So there was some syntax error .Change this and restart the instance. It should work

0 Karma

neiowe
Path Finder

@mayurr98, thank you very much for all your assistance! You original solution worked perfect, I just had the wrong sourcetype in the transforms as pointed out by @micahkemp below. Thanks to you both for helping be resolve this.

0 Karma

mayurr98
Super Champion

Hey @neiowe

Glad to help you!! Pls upvote as well

0 Karma

micahkemp
Champion

The class name of the transforms setting should not matter.

0 Karma

mayurr98
Super Champion

Try
index=your_index “10.3.30.35”

Do a real time search and see if you getting events

0 Karma

neiowe
Path Finder

Yes, I am still get real-time events. Just a side questions: Does a change to transform.conf require a restart of Splunk? I have been doing it but just want to confirm that it is necessary.

0 Karma

mayurr98
Super Champion

Yes restart is must.It is given in doc as well

Refer this doc to exactly same as mentioned in this doc
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad

Look for Discard specific events and keep the rest in Filter event data and send to queues.

Keep all the naming convention same as given in doc and try it must work as i have tried and worked 100%

I hope this solves your problem !

0 Karma

neiowe
Path Finder

This doesn't appear to be working for me. I tried for just the one IP address first. Here's what I added to the transforms/props on my indexers. I also restarted Splunk on the indexers. What am I missing?

In transforms.conf

[palo_setnull]
REGEX = 10.3.30.35
DEST_KEY = queue
FORMAT = nullQueue

In props.conf

[pan:traffic]
TRANSFORMS-set = palo_setnull

0 Karma

mayurr98
Super Champion

Could you please share the sample event of this ip? Cause regex should match the sample event

Also , could you please keep just setnull instead of palo_setnull and restart the instance?

Also palo:traffic is the name of your sourcetype?

0 Karma

neiowe
Path Finder

Jan 5 15:11:18 ISG-5020-FW1 1,2018/01/05 15:11:17,007701002549,TRAFFIC,drop,1,2018/01/05 15:11:17,10.3.30.35,172.31.155.128,0.0.0.0,0.0.0.0,Rule 43,aca\sa_qualsys,,not-applicable,vsys1,inside,DMZ31,ethernet1/2,,ISG-LOG,2018/01/05 15:11:17,0,1,7712,12374,0,0,0x4000,udp,deny,60,60,0,1,2018/01/05 15:11:18,0,any,0,1892346490,0x0,10.0.0.0-10.255.255.255,172.16.0.0-172.31.255.255,0,1,0,policy-deny,12,0,0,0,,ISG-5020-FW1,from-policy

I already have a setnull but I can change to setnull2 if you think it will make a difference.

0 Karma

mayurr98
Super Champion

Put 10\.3\.30\.35
Also one more thing, are you looking historical events or real time events cause this exercise will exclude future events and not the historical events so look for real time events for this ip.

0 Karma

neiowe
Path Finder

I actually do have 10.3.30.35 but for some reason it didn't show up above when I copied the editor. I am looking at real-time events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...