topic Re: Filter Events per EventID in Getting Data In

Filter Events per EventID

mlb19 — Thu, 13 Aug 2015 10:05:07 GMT

Hi Splunkers,

I am relatively new to splunk so I just have an basic knowledge and I apologize if my question is answered else where due a lack of knowledge (even tough I looked quite a while for a solution ).

Most solutions are for older Splunk versions and did not work.

What I want to do:

I want to filter specific events by an EventID (like Windows event log but I also have different logs so I would appreciate a solution where I only have to change the sourcetype and IDs.

From other Questions I have the following:

index.conf:

[monitor://<path>\20150809]
sourcetype = CDE0
blacklist = 5156
disabled = 0

that blacklist was my first try but I think I can only black list files and folders in there.

props.conf:

[CDE0]
TRANSFORMS-t1=eliminate-eventcodes

transforms.conf:

[eliminate-eventcodes]
REGEX=EventID=(5156)
DEST_KEY=queue
FORMAT=nullQueue

I think my problem is in the transforms.conf but I am not really sure.

EDIT: It might be obvious but the filtering should happen before indexing

Thank you

Re: Filter Events per EventID

s2_splunk — Thu, 13 Aug 2015 20:33:21 GMT

For Windows events pulled with a WinEventLog:xxxx stanza in your inputs.conf, you can use native filtering as described here
Detailed documentation is available here

If you specify "blacklist" underneath a [monitor:] stanza, the RegEx will be applied to the , not the events in the log file.
Please read up here on how to use props/transforms to filter events from non-Windows eventlog sources.

Re: Filter Events per EventID

hgrow — Fri, 14 Aug 2015 07:01:45 GMT

Hi mlb19,

additionally to the documentation ssievert posted a sample event of the data you don't want to index would be helpful.

You are right with your inputs.conf (i assume thats a typo). The blacklisting/whitelisting works on files or some inputs like the windows events ssievert mentioned. Otherwise your props.conf and transforms.conf dont seem to be absolutely off.

The guess is, thats your REGEX=EventID=(5156) is just not matching your events.

Greetings

Re: Filter Events per EventID

mlb19 — Fri, 14 Aug 2015 08:27:55 GMT

Okay, thank you. I will try that.
I guess you meant files or folders when you said "RegEx will be applied to the ," sice that is what I found while I was browsing through the documentation.

Re: Filter Events per EventID

mlb19 — Fri, 14 Aug 2015 08:37:35 GMT

Hi hgrow,

yes that was just a typo 🙂 I will edit it now.

Here are some examples of Events I want to filter:

Some of them have an LogCode or an specific ID:

14:32:59.645 | LogCode:046 | AppName: | ProcessID:16778070(854) | Msg: ERR. File - Read Line - Unexpected end of file.

But some others only have text:

INFO [2015/07/15 00:07:29.681] C1| Using primary system database, via direct ODBC.

For the post I want to filter that event:

8/8/2015 23:46:27 8 5156 Microsoft-Windows-Security-Auditing The Windows Filtering Platform has allowed a connection. Application Information:

Thanks

--
I can't edit my original post because I exceeded my limit of 2 posts per day...

Re: Filter Events per EventID

hgrow — Fri, 14 Aug 2015 09:20:37 GMT

Hi mlb19,

it's your REGEX thats not match your Event you want to filter. Despite the filtering of windows event what generally happens is, that the REGEX is just looking for a String in your event, that is matching the regex.

If you are working with regex https://regex101.com is always a great tool. Just insert your events into the test string field and try your regex.

For your specific events 5156 is the simplest Regex that would match the event. But every other event that might contain the numerical sequence 5156 is also filtered. A more precise one could be something like :

\d*\/\d*\/\d*\s\d*\:\d*\:\d*\s\d*\s5156

EDIT: less accurate but more optimized regex

(\d+.){6}\d+\s5156

Hope it helps

Re: Filter Events per EventID

mlb19 — Tue, 18 Aug 2015 16:13:40 GMT

Hi hgrow,

I tried all of your regex but none of them works.
This is how my .config files look like in \etc\system\local:

inputs.conf:
[monitor://\Eventvwr1599sec.txt]
sourcetype = cde0

crcSalt = \Eventvwr1599sec.txt
disabled = 0

props.conf:
[source::>\Eventvwr1599sec.txt]

transforms.conf:
[filter5156]
REGEX = (\d+.){6}\d+\s5156
DEST_KEY = queue
FORMAT = nullQueue
TRANSFORMS-cde0 = filter5156

I guess I did a very stupid mistake somewhere...
If I look trough my indexed events sourcetype=cde0 5156 works fine.

Any idea what I am doing wrong?

Re: Filter Events per EventID

hgrow — Tue, 18 Aug 2015 18:50:14 GMT

Hi mlb19
You left something out what've already done right in your initial configuration. In your propfs.conf you forgot the TRANSFORMS- attribute to determine the routing. The next this is [source::>Eventvwr1599sec.txt] looks a bit off. I'm not sure if thats a typo or not but the generall form is:

[<spec>]
TRANSFORMS-routing=<transforms_stanza_name>

Where can be:

sourcetype, the source type of an event
host::host, where is the host for an event
source::source, where source is the source for an event

Source: Configure routing

To make it easy i would go with the sourcetype here:
propfs.conf

[cde0]
TRANSFORMS-filterrouting=filter5156

Other than that it looks good to me. The REGEX is not quite correct, but i assume Splunk-Answers is not showing the escape Symbols \ correctly. It should be

REGEX = (\d+.){6}\d+\s5156

Greetings

Re: Filter Events per EventID

mlb19 — Wed, 19 Aug 2015 08:43:02 GMT

Oh yeah! It finally works! Thank you very much hgrow!

Re: Filter Events per EventID

hgrow — Wed, 19 Aug 2015 22:39:20 GMT

Hi mlb19,

it's nice to hear 🙂 Good luck with your future splunking !