Getting Data In

Filtering events using NullQueue

Contributor

I was wondering if there is any way to filter eventcodes, but not every event that is being passed through. For example is there a way to block EventCode 4624, but just the debug messages and let the rest pass?

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

1 Solution

Legend

[Updated to show that you can do multiple transforms]

Okay - given the answer from Kristian about the type, I think I can show you how to filter the events. Assuming that the sourcetype is called WinEventLog:Security...

props.conf

[WinEventLog:Security]
TRANSFORMS-t1=eliminate-4624-debug
TRANSFORMS-t2=eliminate-eventcodes

transforms.conf

[eliminate-4624-debug]
REGEX=(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s
DEST_KEY=queue
FORMAT=nullQueue

[eliminate-eventcodes]
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)
DEST_KEY=queue
FORMAT=nullQueue

Now, this is not the tightest regular expression, so I would test it with the following search:

sourcetype="WinEventLog:Security" 
| regex _raw="(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s"

If this search matches only the data that you want to eliminate, then great. Otherwise, I may still need to see a sample of the data...

View solution in original post

Path Finder

Hi Guys,

can i do the same as : REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

if the EventCode is a field i created in an extract field ?

0 Karma

Legend

[Updated to show that you can do multiple transforms]

Okay - given the answer from Kristian about the type, I think I can show you how to filter the events. Assuming that the sourcetype is called WinEventLog:Security...

props.conf

[WinEventLog:Security]
TRANSFORMS-t1=eliminate-4624-debug
TRANSFORMS-t2=eliminate-eventcodes

transforms.conf

[eliminate-4624-debug]
REGEX=(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s
DEST_KEY=queue
FORMAT=nullQueue

[eliminate-eventcodes]
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)
DEST_KEY=queue
FORMAT=nullQueue

Now, this is not the tightest regular expression, so I would test it with the following search:

sourcetype="WinEventLog:Security" 
| regex _raw="(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s"

If this search matches only the data that you want to eliminate, then great. Otherwise, I may still need to see a sample of the data...

View solution in original post

Legend

@erstexas - it depends. On a Universal Forwarder, no. On a heavy forwarder, yes, you can place the transforms.conf and the props.conf on the forwarder.

However, Splunk generally recommends that you use a Universal Forwarder and do this parsing on the indexers. This keeps the processing load low on the production server that is running the forwarder. If you are thinking that you want to limit the network traffic, good idea but - experience says that it isn't worth the trouble unless you will be eliminating more than 50% of the events before forwarding.

0 Karma

Path Finder

Can this be placed on the servers that are running the Forwarder? I would rather have it not sent to the Indexer at all. Or maybe that is what is implied?

0 Karma

Legend

Michael - I would suggest that you have different stanzas in props.conf that invoke different stanzas in transforms.conf

All of the stanzas could send data to the nullqueue but each would have a different regex. Even if there is a way to combine them, I would probably keep them separate for clarity.

0 Karma

Motivator

I am curious how you would change using nullqueue/blacklist to the more common way of doing it where you have a "pass" transform with a whitelist and nullqueue anything else, but still allow for the special case he brought up here where you want to drop the debug 4624 events.

0 Karma

Contributor

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

0 Karma

Contributor

I believe this is exactly what i am looking for. We were just trying to use the debug messages as an example to get the concept. I will test it out next week and let you know. I thank you very much.

0 Karma

Ultra Champion

well, I made an assumption regarding the Type=Debug... I'd also like sample data...

/k

0 Karma

Ultra Champion

If you by debug mean the Type=Debug (I don't know if it exists, I only have 'Informational' in my logs). Therefore I used ComputerName in the example below.

The following regex works with rex inline in the search. It should probably work with the instructions in http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Discard_specific_events...

sourcetype=wineventlog:security EventCode="4624" | rex ".*(?<blaha>EventCode=4624[\n\r\w\.=]*ComputerName=some.host.name).*"

Hope this helps,

Kristian

Contributor

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

0 Karma

Contributor

Thank you for your help Kristian.

0 Karma

Legend

To use the nullQueue, you must be able to write a regular expression that identifies the events to be eliminated.

For the event code, that would be

EventCode\s*=\s*4624

but I am not sure how you would identify this as a debug message. Can you post an example of a few events?

Ultra Champion

Yes you can have multiple transforms that send stuff to the null queue;

props.conf
[sourcetype_x]
TRANSFORMS-delete_stuff = drop_a, drop_b

transforms.conf
[drop_a]
REGEX = a
DEST = queue
FORMAT = nullQueue

[drop_b]
REGEX = b
DEST = queue
FORMAT = nullQueue

That's the same as having REGEX = a|b in one nullQueue transform.

0 Karma

Communicator

What would my REGEX line in the transforms.conf be to ELIMINATE any events that don't have this string? I must be missing something. I only want to ingest events that have this string at the beginning of the line:  "|>>>>>>|" 

In REGEX that should be ^\|>>>>>>\| right? 

So how to i set the transforms.conf REGEX= line to say anything that doesn't match the above REGEX, drop to the nullqueue?

 

Thanks in advance!!!

Joe

0 Karma

Contributor

I dont know if that is exactly what i was looking for. I probably worded the question in a confusing way.

Heres another example:
we want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues? AND if so how would we do this?

0 Karma