topic Re: How to write the regex to filter events by contents of a specific field in transforms.conf? in Splunk Search

How to write the regex to filter events by contents of a specific field in transforms.conf?

pjohnson1 — Tue, 10 Nov 2015 01:53:48 GMT

I am creating a filter to only keep certain events which contain a specific country code (they are actually hostnames which contain the country code).

props.conf

[log*]
TRANSFORMS-keep-LOG = setnull,keep-LOG_transform

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep-LOG_transform]
REGEX = SERVER01:\s+(KR|SG|IN|PH|TW|TH)
DEST_KEY = queue
FORMAT = indexQueue

How can I create a REGEX on a specific field?

Field extraction is complete with this data source, but I would like to filter all events with KR,SG,IN,PH,TW,TH in a specific field like host.

Thanks.

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

pjohnson1 — Tue, 10 Nov 2015 02:58:33 GMT

I referenced this https://answers.splunk.com/answers/47982/extracting-field-from-a-field-other-than-raw-in-props-conf.html

[keep-LOG_transform]
SOURCE_KEY = host
REGEX = (KR|SG|IN|PH|TW|TH)
DEST_KEY = queue
FORMAT = indexQueue

But still no joy. Any guidance please...

Thanks.

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

jeffland — Tue, 10 Nov 2015 10:15:35 GMT

Not sure if it's a typo in your question or in your settings, but there is a space in your transforms stanza name which is not there in your props.conf.

Where did you place these settings in your environment? It should be either directly on your indexer or on a heavy forwarder, depending on where your events are parsed.

Also, how does the setting fail? Is every event still indexed regardless of country code, or are none of them?

My guess is that these settings (even when applied properly) don't work for you because the transform is applied to _raw, and your raw data either doesn't contain SERVER01:, or that all of them do (assuming that SERVER01 is the value of your host field). So can we please see a small sample of an event? (Mask any sensitive data).

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

krish3 — Tue, 10 Nov 2015 10:19:56 GMT

Please check this link and let me know how it goes.

Click Me

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

pjohnson1 — Tue, 10 Nov 2015 10:52:20 GMT

Thanks for the input...

This is on the indexer under ..\system\local\ and it indexes everything and not filtering by the country codes.

Nov 10 10:42:23 SymantecServer SERVER01: KR-PC-098763,Continue,,File Read
Nov 10 14:22:23 SymantecServer SERVER01: CN-PC-012345,Continue,,File Read
Nov 10 15:32:23 SymantecServer SERVER01: SG-PC-054323,Continue,,File Read

I tested with the following and it appeared to work fine.
index=av | regex _raw="SERVER01:\s+(KR|SG|IN|PH|TW|TH)"

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

jeffland — Tue, 10 Nov 2015 17:27:16 GMT

If that's the raw data, that should work, so it would have to be something before actually applying the regex that isn't working as indended in that case. I'm curious, if you leave out the keep-LOG_transform stanza in the list of transforms applied in your props.conf, are events still indexed? That would mean changing your props.conf to

[log*]
TRANSFORMS-keep-LOG = setnull

If yes, then these settings aren't applied to the sourcetype at all. What exactly is your sourcetype named? Is using an asterisk in your stanza required, or could you temporarily use the precise sourcetype name to see if that helps? Also, is there another setting that may override this setting (settings for sourcetype are overridden by settings for host:: and source::)?

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

pjohnson1 — Tue, 10 Nov 2015 23:09:00 GMT

The events are still indexed if just using TRANSFORMS-keep-LOG = setnull.

I have changed to the precise sourcetype and check via search but events are still being indexed.

[host:123.456.789]

and

[sep12:behavior]
[sep12:traffic]
[sep12:agt_system]
[sep12:scan]    
[sep12:risk]    
[sep12:log]

I have another props/transform which works perfectly for something else but this one is baffelling...

Re: How to write the regex to filter events by contents of a specific field in transforms.conf?

jeffland — Fri, 20 Nov 2015 10:18:09 GMT

Hi, sorry for taking so long to come back to you.
I see you are using [host:123], which I believe is supposed to be [host::123]. And the other stanzas, is your sourcetype sep12:behavior, and other settings are applied properly to this stanza?