topic Using REGEX with multiple parameters in the same transforms.conf instance in Splunk Enterprise

Using REGEX with multiple parameters in the same transforms.conf instance

AndrewF17 — Wed, 28 Aug 2024 15:00:15 GMT

Hello everyone,

I'm trying to filter out some logs in the IA-WindowsSecurity Application.

The indexed values are when:

- The EventCode=4634 AND the Security_ID="*$"

I created an app deployed on an index with the following props and transforms config:

Props.conf

[WinEventLog] TRANSFORMS-remove_computer_logoff = remove_logoff

Transforms.conf

[remove_logoff] REGEX = DEST_KEY = queue FORMAT = nullQueue I made the following regex for matching the event: - EventCode=4634 - Security_ID=".*\$$"

I'm not sure how to correctly "put together" these two REGEXES.

I did a lot of testing with different types of regexes (in PCRE Format), but I wasn't able to make it work.

Can someone please help me?

Thanks in advance

Re: Using REGEX with multiple parameters in the same transforms.conf instance

richgalloway — Tue, 27 Aug 2024 14:19:27 GMT

Try putting the two expressions together separated by [\s\S]+ to represent any intervening text.

EventCode=4634[\s\S]+Security_ID=".+?\$$"

If the order of fields might vary, use this variation to match both orders.

(?:EventCode=4634[\s\S]+Security_ID=".+?\$$")|(?:Security_ID=".+?\$$"[\s\S]+EventCode=4634)

Re: Using REGEX with multiple parameters in the same transforms.conf instance

PickleRick — Tue, 27 Aug 2024 15:18:05 GMT

One way is to do it as @richgalloway showed - with a composite regex accounting for both orders of fields (Just include possible whitespaces - I don't remember if they are included in windows events or not).

Another way is to use INGEST_EVAL and use something like this for your eval

queue=if(match(first_regex_and_so_on) AND match(second_regex...), "nullQueue", queue)

Be aware thought that it won't work for the events from inputs with renderXml=true.

Anyway, additionally you could look into filtering out those values even earlier - in your forwarder's input's stanza using blacklisting.

Re: Using REGEX with multiple parameters in the same transforms.conf instance

AndrewF17 — Wed, 28 Aug 2024 10:55:45 GMT

Hi,

Thanks for your help.

I tried the following configuration in my transforms.conf:

[remove_logoff] INGEST_EVAL = queue=if(match(_raw,"EventCode=4634") AND match(_raw,"Security\sID:[\s]+.*\$"), "nullQueue", queue)

props.conf

[WinEventLog] TRANSFORMS-remove_computer_logoff = remove_logoff

But after I run the query, I still get the unwanted logs. I tried to make the query on the search as well to check if the regex were right and everything seems fine.

index=* sourcetype=WinEventLog | eval result=if(match(_raw,"EventCode=4634") AND match(_raw,"Security\sID:[\s]+.*\$"), "Filter", "No need to filter this log") | stats count by host, result

Am I missing something?

P.S. I cannot do a blacklist directly on the hosts

Re: Using REGEX with multiple parameters in the same transforms.conf instance

PickleRick — Wed, 28 Aug 2024 11:59:00 GMT

AFAIR I had mixed results with transform not containing anything in the REGEX field. Try to explicitly add

REGEX = .

to match anything to the transform.

Re: Using REGEX with multiple parameters in the same transforms.conf instance

AndrewF17 — Wed, 28 Aug 2024 13:10:43 GMT

I tried this conf:

[remove_logoff] REGEX = "(?:EventCode=4634)" INGEST_EVAL = queue=if(match(_raw,"Security\sID:[\s]+.*\$"), "nullQueue", queue)

and also with

REGEX = .

But in both cases I'm still getting logs.

Re: Using REGEX with multiple parameters in the same transforms.conf instance

PickleRick — Wed, 28 Aug 2024 13:16:23 GMT

OK. I'd try to verify whether the transform is called at all. I have a feeling that it is not for some reason.

You can for testing create some "sure fire" transform and check if it is being applied.

Are you sure you're doing it on the right component?

Re: Using REGEX with multiple parameters in the same transforms.conf instance

AndrewF17 — Wed, 28 Aug 2024 13:27:25 GMT

Unfortunately, in both cases doesn't work.

I tried also to work with the raw logs on regex101 and I came up with this regex:

EventCode=4634+[^$]+Security ID:\s+.*\$

But I still getting logs.