Using REGEX with multiple parameters in the same transforms.conf instance

[AndrewF17]

Hello everyone,

I'm trying to filter out some logs in the IA-WindowsSecurity Application.

The indexed values are when:

- The EventCode=4634 AND the Security_ID="*$"

I created an app deployed on an index with the following props and transforms config:

Props.conf

[WinEventLog]
TRANSFORMS-remove_computer_logoff = remove_logoff

Transforms.conf

[remove_logoff]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

I made the following regex for matching the event:

- EventCode=4634

- Security_ID=".*\$$"

I'm not sure how to correctly "put together" these two REGEXES.

I did a lot of testing with different types of regexes (in PCRE Format), but I wasn't able to make it work.

Can someone please help me?

Thanks in advance

[PickleRick]

One way is to do it as @richgalloway showed - with a composite regex accounting for both orders of fields (Just include possible whitespaces - I don't remember if they are included in windows events or not).

Another way is to use INGEST_EVAL and use something like this for your eval

queue=if(match(first_regex_and_so_on) AND match(second_regex...), "nullQueue", queue)

Be aware thought that it won't work for the events from inputs with renderXml=true.

Anyway, additionally you could look into filtering out those values even earlier - in your forwarder's input's stanza using blacklisting.

[AndrewF17]

Hi,

Thanks for your help.

I tried the following configuration in my transforms.conf:

[remove_logoff]
INGEST_EVAL = queue=if(match(_raw,"EventCode=4634") AND match(_raw,"Security\sID:[\s]+.*\$"), "nullQueue", queue)

props.conf

[WinEventLog]
TRANSFORMS-remove_computer_logoff = remove_logoff

 But after I run the query, I still get the unwanted logs. I tried to make the query on the search as well to check if the regex were right and everything seems fine.

index=* sourcetype=WinEventLog
| eval result=if(match(_raw,"EventCode=4634") AND match(_raw,"Security\sID:[\s]+.*\$"), "Filter", "No need to filter this log")
| stats count by host, result

Am I missing something?

P.S. I cannot do a blacklist directly on the hosts

[PickleRick]

AFAIR I had mixed results with transform not containing anything in the REGEX field. Try to explicitly add

REGEX = .

to match anything to the transform.

[AndrewF17]

I tried this conf:

[remove_logoff]
REGEX = "(?:EventCode=4634)"
INGEST_EVAL = queue=if(match(_raw,"Security\sID:[\s]+.*\$"), "nullQueue", queue)

and also with

REGEX = .

But in both cases I'm still getting logs.

[PickleRick]

OK. I'd try to verify whether the transform is called at all. I have a feeling that it is not for some reason.

You can for testing create some "sure fire" transform and check if it is being applied.

Are you sure you're doing it on the right component?

[richgalloway]

Try putting the two expressions together separated by [\s\S]+ to represent any intervening text.

EventCode=4634[\s\S]+Security_ID=".+?\$$"

If the order of fields might vary, use this variation to match both orders.

(?:EventCode=4634[\s\S]+Security_ID=".+?\$$")|(?:Security_ID=".+?\$$"[\s\S]+EventCode=4634)

[AndrewF17]

Unfortunately, in both cases doesn't work.

I tried also to work with the raw logs on regex101 and I came up with this regex:

EventCode=4634+[^$]+Security ID:\s+.*\$

But I still getting logs.