topic Re: Filtering events using NullQueue in Getting Data In

Filtering events using NullQueue

Michael_Schyma1 — Thu, 20 Sep 2012 18:37:03 GMT

I was wondering if there is any way to filter eventcodes, but not every event that is being passed through. For example is there a way to block EventCode 4624, but just the debug messages and let the rest pass?

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

Re: Filtering events using NullQueue

lguinn2 — Thu, 20 Sep 2012 19:42:52 GMT

To use the nullQueue, you must be able to write a regular expression that identifies the events to be eliminated.

For the event code, that would be

EventCode\s*=\s*4624

but I am not sure how you would identify this as a debug message. Can you post an example of a few events?

Re: Filtering events using NullQueue

kristian_kolb — Mon, 28 Sep 2020 12:29:11 GMT

If you by debug mean the Type=Debug (I don't know if it exists, I only have 'Informational' in my logs). Therefore I used ComputerName in the example below.

The following regex works with rex inline in the search. It should probably work with the instructions in http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

sourcetype=wineventlog:security EventCode="4624" | rex ".*(?<blaha>EventCode=4624[\n\r\w\.=]*ComputerName=some.host.name).*"

Hope this helps,

Kristian

Re: Filtering events using NullQueue

lguinn2 — Thu, 20 Sep 2012 20:46:20 GMT

[Updated to show that you can do multiple transforms]

Okay - given the answer from Kristian about the type, I think I can show you how to filter the events. Assuming that the sourcetype is called WinEventLog:Security...

props.conf

[WinEventLog:Security]
TRANSFORMS-t1=eliminate-4624-debug
TRANSFORMS-t2=eliminate-eventcodes

transforms.conf

[eliminate-4624-debug]
REGEX=(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s
DEST_KEY=queue
FORMAT=nullQueue

[eliminate-eventcodes]
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)
DEST_KEY=queue
FORMAT=nullQueue

Now, this is not the tightest regular expression, so I would test it with the following search:

sourcetype="WinEventLog:Security" 
| regex _raw="(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s"

If this search matches only the data that you want to eliminate, then great. Otherwise, I may still need to see a sample of the data...

Re: Filtering events using NullQueue

kristian_kolb — Thu, 20 Sep 2012 21:05:37 GMT

well, I made an assumption regarding the Type=Debug... I'd also like sample data...

Re: Filtering events using NullQueue

Michael_Schyma1 — Thu, 20 Sep 2012 21:35:55 GMT

I dont know if that is exactly what i was looking for. I probably worded the question in a confusing way.

Heres another example:
we want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues? AND if so how would we do this?

Re: Filtering events using NullQueue

Michael_Schyma1 — Fri, 21 Sep 2012 13:09:19 GMT

I believe this is exactly what i am looking for. We were just trying to use the debug messages as an example to get the concept. I will test it out next week and let you know. I thank you very much.

Re: Filtering events using NullQueue

Michael_Schyma1 — Fri, 21 Sep 2012 13:09:50 GMT

Thank you for your help Kristian.

Re: Filtering events using NullQueue

kristian_kolb — Mon, 28 Sep 2020 12:29:31 GMT

Yes you can have multiple transforms that send stuff to the null queue;

props.conf
[sourcetype_x] TRANSFORMS-delete_stuff = drop_a, drop_b
transforms.conf
[drop_a] REGEX = a DEST = queue FORMAT = nullQueue

[drop_b]
REGEX = b
DEST = queue
FORMAT = nullQueue

That's the same as having REGEX = a|b in one nullQueue transform.

Re: Filtering events using NullQueue

Michael_Schyma1 — Fri, 21 Sep 2012 13:32:21 GMT

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

Re: Filtering events using NullQueue

Michael_Schyma1 — Fri, 21 Sep 2012 13:32:30 GMT

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

Re: Filtering events using NullQueue

wrangler2x — Fri, 15 Feb 2013 22:17:03 GMT

I am curious how you would change using nullqueue/blacklist to the more common way of doing it where you have a "pass" transform with a whitelist and nullqueue anything else, but still allow for the special case he brought up here where you want to drop the debug 4624 events.

Re: Filtering events using NullQueue

lguinn2 — Mon, 18 Feb 2013 19:03:39 GMT

Michael - I would suggest that you have different stanzas in props.conf that invoke different stanzas in transforms.conf

All of the stanzas could send data to the nullqueue but each would have a different regex. Even if there is a way to combine them, I would probably keep them separate for clarity.

Re: Filtering events using NullQueue

erstexas — Fri, 21 Jun 2013 20:19:52 GMT

Can this be placed on the servers that are running the Forwarder? I would rather have it not sent to the Indexer at all. Or maybe that is what is implied?

Re: Filtering events using NullQueue

lguinn2 — Fri, 21 Jun 2013 23:19:30 GMT

@erstexas - it depends. On a Universal Forwarder, no. On a heavy forwarder, yes, you can place the transforms.conf and the props.conf on the forwarder.

However, Splunk generally recommends that you use a Universal Forwarder and do this parsing on the indexers. This keeps the processing load low on the production server that is running the forwarder. If you are thinking that you want to limit the network traffic, good idea but - experience says that it isn't worth the trouble unless you will be eliminating more than 50% of the events before forwarding.

Re: Filtering events using NullQueue

shayfa — Wed, 11 Nov 2015 09:34:12 GMT

Hi Guys,

can i do the same as : REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

if the EventCode is a field i created in an extract field ?

Re: Filtering events using NullQueue

joesrepsolc — Fri, 03 Jul 2020 19:50:43 GMT

What would my REGEX line in the transforms.conf be to ELIMINATE any events that don't have this string? I must be missing something. I only want to ingest events that have this string at the beginning of the line: "|>>>>>>|"

In REGEX that should be ^\|>>>>>>\| right?

So how to i set the transforms.conf REGEX= line to say anything that doesn't match the above REGEX, drop to the nullqueue?

Thanks in advance!!!

Joe

Re: Filtering events using NullQueue

jotne — Mon, 30 Jan 2023 07:52:30 GMT

Some comments to the above post.

Its better to remove stuff at the Universal Forwarder instead of HF or Index.

So to remove 4662, add the following to an input.file

# Used to block 4662 message

[WinEventLog://Security]
blacklist1 = 4662

Or you can do like this. Block all 4662 message except 4662 with Message="ms-Mcs-AdmPwd"

[WinEventLog://Security]
whitelist1 = EventCode="^4662$" Message="ms-Mcs-AdmPwd"
whitelist2 = EventCode="^((?!4662$)[0-9]*)$"

Take care with this:

REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

This may block all, due to the double ||, I gess that is a typo.
Also it will block 1552, 5525 etc, so here you should use ^ and $