topic Re: Heavy Forwarder REGEX Filting Issues in Splunk Search

Heavy Forwarder REGEX Filting Issues

dscoland — Fri, 21 Mar 2014 17:38:29 GMT

Hi All,

I am having difficulty finding in-depth documentation on REGEX syntax, and I am attempting to filter out [WinEventLog:Security] logs from our central Splunk Instance by use of a heavy forwarder.

On our Heavy Forwarder, we have the following filters setup:

props.conf

[WinEventLog:Security]
TRANSFORMS-security= npm-setnull, everythingElse

Filter NPM account Security logs to nullQueue

[npm-setnull]
REGEX = (?msi)Account_Name="ACCOUNT@DOMAIN.COM"
DEST_KEY = queue
FORMAT = nullQueue

Send everything else to be indexed in the indexQueue

[everythingElse]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

This is currently not filtering any of the logs, and I feel thate the syntax is incorrect, but I can't find any solid documentation on how to format this. Can anyone help?

Thank you in advance,r
Daniel

Re: Heavy Forwarder REGEX Filting Issues

kristian_kolb — Fri, 21 Mar 2014 21:12:02 GMT

You'll need to reverse the order in which the transforms are called from props.conf. Or in your case, just remove the [everythingElse] stanza and the reference to it. While it might seem a bit contra-intuitive, think of it like this;

"All events will pass through all transforms in the specified order before being returned for further processing"

It is not like there is some kind of 'break-on-match' processing. Thus, first setting the queue to nullQueue for some events, and then immediately resetting it back to indexQueue (for all events), will have the same results as not having any filtering at all.

UPDATE:

Yes, wolverine is right. The field name Account_Name is a so-called 'cleaned' name, i.e, spaces are replaced with underscores etc. The REGEX must match the actual text in the event, i.e.;

REGEX = Account\sName:\s+ACCOUNT@DOMAIN.COM

Re: Heavy Forwarder REGEX Filting Issues

dscoland — Mon, 28 Sep 2020 16:12:29 GMT

Hi Kristian,

Thank you for your response.

I now have:

[WinEventLog:Security]
TRANSFORMS-security= npm-setnull

[npm-setnull]
REGEX = (?msi)Account_Name="NPM@EDM.LOCAL"
DEST_KEY = queue
FORMAT = nullQueue

It still appears not to be filtering. Do I need to restart the Heavy Forwarder instance?

Daniel

Re: Heavy Forwarder REGEX Filting Issues

the_wolverine — Sat, 22 Mar 2014 00:44:38 GMT

I think the issue might be with your REGEX:

REGEX = (?m)Account_Name="ACCOUNT@DOMAIN.COM"

Re: Heavy Forwarder REGEX Filting Issues

kristian_kolb — Sun, 23 Mar 2014 21:40:38 GMT

good spotting, wolverine! Updated my answer to reflect on this as well.

Re: Heavy Forwarder REGEX Filting Issues

dscoland — Mon, 24 Mar 2014 12:37:26 GMT

Hi wolverine!

Thank you for your response. I am going to try this out. I know I probably sound like a broken record. But does Splunk provide REGEX documentation? I feel rather ignorant not understanding (?msi) and (?m) syntax; or is this provided elsewhere?

Thanks for your help, guys.

Daniel

Re: Heavy Forwarder REGEX Filting Issues

kristian_kolb — Mon, 24 Mar 2014 12:45:14 GMT

Hi dscoland,

www.regular-expressions.info

regexr.com

are two commonly referenced guides for learning/testing regular expressions.

Re: Heavy Forwarder REGEX Filting Issues

dscoland — Mon, 24 Mar 2014 14:18:22 GMT

Awesome, thanks Kristian!

As it turns out, this did not fix the issue. I will have to dig deeper in this.

Thanks for your continued help, guys.
Daniel

Re: Heavy Forwarder REGEX Filting Issues

dscoland — Mon, 24 Mar 2014 16:40:36 GMT

Thanks guys, I managed to get the filter working.

I changed the field to look for the Logon Account

[npm-setnull] REGEX = Logon\sAccount:\snpm DEST_KEY = queue FORMAT = nullQueue

But I believe the Splunkd service in Services.msc needs to be rebooted in order to start the filter.

Regards!
Daniel

Re: Heavy Forwarder REGEX Filting Issues

kristian_kolb — Mon, 24 Mar 2014 19:46:33 GMT

Yes, you need to restart splunkd - either from the outside, like restarting the service, or from the inside, in the GUI, or hit the refresh endpoint url;

http(s)://your_splunk:8000/en-US/debug/refresh

That will also reload most configs.

Purely search-related configs, such as field extractions will be reloaded for each search.