topic Re: Help with props and transforms in Getting Data In

Help with props and transforms

a212830 — Mon, 13 May 2019 14:25:46 GMT

Hi,

I have a feed where the fields are separated by brackets (<>). I have a transforms.conf that extracts the fields automatically:

REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

Unfortunately, the fields are all uppercase. I don't see any way to make the fields lowercase, so I've started creating aliases, using FIELD_ALIAS. We need to do this so that they are caught by our ES rules. I also need to do a transforms to map the values appropriately.

Here is a sample field: "Allow

I want to create an alias with the field name to be "action" and a transform that makes the value "allowed". I get the new field, but the transform is not working. Here's what I have configured:

props.conf:

FIELDALIAS-action = ACTION as action

transforms.conf:

[forcepoint_xml]
REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

[ACTION]
REGEX = (Allow|Permit)
FORMAT = ACTION::allowed

Any suggestions?

Re: Help with props and transforms

maciep — Mon, 13 May 2019 15:13:16 GMT

you might be able to just use eval instead of a field alias. Maybe like this...obviously you can additional allowed/blocked logic:

EVAL-action = if(match(ACTION,"(?i)allow|permit"),"allowed","blocked")

You could also create an automatic lookup to lookup ACTION and spit out the various actions as well - maybe not needed here, but that approach can come in handy in general when mapping data to the CIM.

Re: Help with props and transforms

FrankVl — Mon, 13 May 2019 15:13:26 GMT

Not entirely sure why your current transforms is not working (except that you are not showing the respective props.conf line) but this is typically resolved with calculated fields or lookups, rather than transforms.

For example, in props.conf (this also gets rid of the need for a FIELDALIAS):

EVAL-action = case(ACTION="Allow","allowed",ACTION="Permit","allowed")

When the number of options grows, a lookup might be a more suitable approach.

PS: any specific reason you're using a custom regex, rather than setting KV_MODE = xml in props.conf?

Re: Help with props and transforms

a212830 — Mon, 13 May 2019 15:25:25 GMT

It's not proper xml - just bracket seperated fields, so the KV_MODE doesnt' work.

Re: Help with props and transforms

woodcock — Mon, 13 May 2019 22:12:56 GMT

You can convert everything in the raw event to lower-case like this:

SEDCMD-ToLowerCase = s/\(.*\)/\L\1/

Re: Help with props and transforms

a212830 — Tue, 14 May 2019 02:06:19 GMT

I thought about that, but what's the load when doing that? It's not a super busy feed, but not small either.

Re: Help with props and transforms

woodcock — Tue, 14 May 2019 02:17:20 GMT

There's only one way to know: YOLO!

Re: Help with props and transforms

FrankVl — Tue, 14 May 2019 07:13:10 GMT

You might want to make that a bit more specific to only make the keys lowercase, not the values.

Re: Help with props and transforms

sloshburch — Mon, 01 Jul 2019 12:42:20 GMT

Couldn't this all be done with one SED command (building on @maciep's suggestion)? I haven't tested but I imagine something with capture groups and forcing lowercase. This essentially combines what everyone else has said.

SED-yaay = s/>(allow|permit)</\L\1/i

The syntax is prob all wrong but the point is that instead of capturing anything you're capturing the specific terms you wanted, using the /i at the end to do case irrelevant. Using the \L that @woodcock introduced to force case on output.

Re: Help with props and transforms

sloshburch — Mon, 01 Jul 2019 12:43:03 GMT

How about a sample of the _raw with all your corporate stuff removed. That will help us help you.