topic Re: Filter Events before Indexing in Getting Data In

Filter Events before Indexing

mh0712 — Tue, 29 Sep 2020 23:00:39 GMT

I get events from a universal forwarder. If "alertd[123456]: ABC:" be in the event, i would like to index it. All other events can be ignored.

Do you have a solution?

2019-01-23T14:22:45+01:00 host kernel: [123456.789101] ll header: yf:ff:ff:ef:ff:ff:00:00:00:00:88:05:01:00
2019-01-23T14:22:49+01:00 host alertd[456789]: get_db_c(): ......
2019-01-23T14:22:50+01:00 host alertd[123456]: CEF:0|abcdef|host|....
2019-01-23T14:22:59+01:00 host alertd[456789]: abc_send(): ......

I have tried the following configuration on the Indexer, but it didn't work:

props.conf
[source::C:\Users\test\testsource.log]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (alertd[\d{1,6}]:\s\w{3}:)
DEST_KEY = queue
FORMAT = indexQueue

Thanks for your solutions.

Re: Filter Events before Indexing

richgalloway — Wed, 30 Jan 2019 12:46:03 GMT

Have you tried TRANSFORMS-set = setparsing,setnull?

Re: Filter Events before Indexing

DMohn — Wed, 30 Jan 2019 12:47:47 GMT

I guess you are missing the escape characters for the square brackets here ...

 (alertd\[\d{1,6}\]\:\s\w{3}\:)

Also, you should make sure, every event will go through both transforms, so maybe modify your props like this:

 [source::C:\Users\test\testsource.log]
 TRANSFORMS-set1 = setnull
 TRANSFORMS-set2 = setparsing

Another option would be to use only a setnull transform, and make sure the regex from the setparsing does NOT match. This would be like so:

 ^((?!(alertd\[\d{1,6}\]\:\s\w{3}\:)).)*$

Re: Filter Events before Indexing

vishaltaneja070 — Wed, 30 Jan 2019 14:19:21 GMT

Try this:
[source::C:\Users\test\testsource.log]
TRANSFORMS-set = setnull,setparsing