Splunk Search

Easiest way to exclude ingestion of events for a specific IP address from a SourceType

Path Finder

I am looking to not ingest events from a specific IP address. I have an IP address that once a week generates a LOT of firewall traffic and which is causing me to exceed my license. What's the easiest/best way to exclude these events. The firewall logs are being sent to a SYSLOG server that is running a Universal Forwarder that sends all logs to my Indexer. I am assuming I will need to use the PROPS/TRANSFORM files to send these to the NULL Queue, but not sure of the proper stanza's to accomplish this. Anyone done this and can provide a sample or suggest a better way to exclude these events?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hey
This is done by defining a regex to match the necessary event(s) and send them to nullqueue

Here is a basic example that will drop everything the events that you do not want.
Let suppose you have ip 192.168.10.11 in the event which generate all these firewall logs and you want exclude these events

Then put:
in transforms.conf

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

And in props.conf

 [your_sourcetype]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-null = setnull

Let me know if this helps!

View solution in original post

Super Champion

@neiowe can you try below:
in transforms.conf:

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

in props.conf:

[pan:traffic]
TRANSFORMS-null= setnull

Path Finder

alt text

alt text

0 Karma

Champion

Does your input set sourcetype=pan:traffic, or does it set sourcetype=pan:log, which is then changed to pan:traffic via a TRANSFORM in the Palo Alto TA?

I'm guessing the latter, which means that the transforms you define for the pan:traffic sourcetype will not be run.

When an event traverses is indexed it will do so for the sourcetype defined when it enters the indexing queue, and changing the sourcetype via a transform will not result in the new sourcetype's transforms being run.

Path Finder

Bingo! That was the problem. I checked my inputs.conf and the sourcetype was being set to palo:log. I change the sourcetype to palo:log in the transforms and it's working perfect now. Thank you very much!

0 Karma

SplunkTrust
SplunkTrust

Hey
This is done by defining a regex to match the necessary event(s) and send them to nullqueue

Here is a basic example that will drop everything the events that you do not want.
Let suppose you have ip 192.168.10.11 in the event which generate all these firewall logs and you want exclude these events

Then put:
in transforms.conf

[setnull]
 REGEX = 192\.168\.10\.11
 DEST_KEY = queue
 FORMAT = nullQueue

And in props.conf

 [your_sourcetype]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-null = setnull

Let me know if this helps!

View solution in original post

SplunkTrust
SplunkTrust

Hey

I think according to the documentation its TRANSFORMS-null = setnull
And in your screenshot you had written TRANSFORMS-set = setnull

So there was some syntax error .Change this and restart the instance. It should work

0 Karma

Path Finder

@mayurr98, thank you very much for all your assistance! You original solution worked perfect, I just had the wrong sourcetype in the transforms as pointed out by @micahkemp below. Thanks to you both for helping be resolve this.

0 Karma

SplunkTrust
SplunkTrust

Hey @neiowe

Glad to help you!! Pls upvote as well

0 Karma

Champion

The class name of the transforms setting should not matter.

0 Karma

SplunkTrust
SplunkTrust

Try
index=your_index “10.3.30.35”

Do a real time search and see if you getting events

0 Karma

Path Finder

Yes, I am still get real-time events. Just a side questions: Does a change to transform.conf require a restart of Splunk? I have been doing it but just want to confirm that it is necessary.

0 Karma

SplunkTrust
SplunkTrust

Yes restart is must.It is given in doc as well

Refer this doc to exactly same as mentioned in this doc
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad

Look for Discard specific events and keep the rest in Filter event data and send to queues.

Keep all the naming convention same as given in doc and try it must work as i have tried and worked 100%

I hope this solves your problem !

0 Karma

Path Finder

This doesn't appear to be working for me. I tried for just the one IP address first. Here's what I added to the transforms/props on my indexers. I also restarted Splunk on the indexers. What am I missing?

In transforms.conf

[palosetnull]
REGEX = 10.3.30.35
DEST
KEY = queue
FORMAT = nullQueue

In props.conf

[pan:traffic]
TRANSFORMS-set = palo_setnull

0 Karma

SplunkTrust
SplunkTrust

Could you please share the sample event of this ip? Cause regex should match the sample event

Also , could you please keep just setnull instead of palo_setnull and restart the instance?

Also palo:traffic is the name of your sourcetype?

0 Karma

Path Finder

Jan 5 15:11:18 ISG-5020-FW1 1,2018/01/05 15:11:17,007701002549,TRAFFIC,drop,1,2018/01/05 15:11:17,10.3.30.35,172.31.155.128,0.0.0.0,0.0.0.0,Rule 43,aca\sa_qualsys,,not-applicable,vsys1,inside,DMZ31,ethernet1/2,,ISG-LOG,2018/01/05 15:11:17,0,1,7712,12374,0,0,0x4000,udp,deny,60,60,0,1,2018/01/05 15:11:18,0,any,0,1892346490,0x0,10.0.0.0-10.255.255.255,172.16.0.0-172.31.255.255,0,1,0,policy-deny,12,0,0,0,,ISG-5020-FW1,from-policy

I already have a setnull but I can change to setnull2 if you think it will make a difference.

0 Karma

SplunkTrust
SplunkTrust

Put 10\.3\.30\.35
Also one more thing, are you looking historical events or real time events cause this exercise will exclude future events and not the historical events so look for real time events for this ip.

0 Karma

Path Finder

I actually do have 10.3.30.35 but for some reason it didn't show up above when I copied the editor. I am looking at real-time events.

0 Karma

Path Finder

This doesn't show the \ but they are there.

0 Karma

Path Finder

Thanks. I will give this a try. One additional question: If I was to use add a second IP address range, i.e. 10.10.0.0/16, how would I do that?

0 Karma

SplunkTrust
SplunkTrust

you can create another stanza with different name i.e. setnull1,setnull2,setnull3,etc
in transforms.conf:

[setnull]
  REGEX = 192\.168\.10\.11
  DEST_KEY = queue
  FORMAT = nullQueue

 [setnull1]
  REGEX = write_regex_for_second_range_or_ip
  DEST_KEY = queue
  FORMAT = nullQueue

In props.conf

[your_sourcetype]
  # Transforms must be applied in this order
  # to make sure events are dropped on the
  # floor prior to making their way to the
  # index processor
  TRANSFORMS-set = setnull,setnull1

This is how you can exclude differenct types of events

0 Karma