Help with props and transforms

[a212830]

Hi,

I have a feed where the fields are separated by brackets (<>). I have a transforms.conf that extracts the fields automatically:

REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

Unfortunately, the fields are all uppercase. I don't see any way to make the fields lowercase, so I've started creating aliases, using FIELD_ALIAS. We need to do this so that they are caught by our ES rules. I also need to do a transforms to map the values appropriately.

Here is a sample field: "Allow

I want to create an alias with the field name to be "action" and a transform that makes the value "allowed". I get the new field, but the transform is not working. Here's what I have configured:

props.conf:

FIELDALIAS-action = ACTION as action

transforms.conf:

[forcepoint_xml]
REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

[ACTION]
REGEX = (Allow|Permit)
FORMAT = ACTION::allowed

Any suggestions?

[sloshburch]

How about a sample of the _raw with all your corporate stuff removed. That will help us help you.

[sloshburch]

Couldn't this all be done with one SED command (building on @maciep's suggestion)? I haven't tested but I imagine something with capture groups and forcing lowercase. This essentially combines what everyone else has said.

SED-yaay = s/>(allow|permit)</\L\1/i

The syntax is prob all wrong but the point is that instead of capturing anything you're capturing the specific terms you wanted, using the /i at the end to do case irrelevant. Using the \L that @woodcock introduced to force case on output.

[woodcock]

You can convert everything in the raw event to lower-case like this:

SEDCMD-ToLowerCase = s/\(.*\)/\L\1/

[FrankVl]

You might want to make that a bit more specific to only make the keys lowercase, not the values.

[a212830]

I thought about that, but what's the load when doing that? It's not a super busy feed, but not small either.

[woodcock]

There's only one way to know: YOLO!

[FrankVl]

Not entirely sure why your current transforms is not working (except that you are not showing the respective props.conf line) but this is typically resolved with calculated fields or lookups, rather than transforms.

For example, in props.conf (this also gets rid of the need for a FIELDALIAS):

EVAL-action = case(ACTION="Allow","allowed",ACTION="Permit","allowed")

When the number of options grows, a lookup might be a more suitable approach.

PS: any specific reason you're using a custom regex, rather than setting KV_MODE = xml in props.conf?

[a212830]

It's not proper xml - just bracket seperated fields, so the KV_MODE doesnt' work.

[maciep]

you might be able to just use eval instead of a field alias. Maybe like this...obviously you can additional allowed/blocked logic:

EVAL-action = if(match(ACTION,"(?i)allow|permit"),"allowed","blocked")

You could also create an automatic lookup to lookup ACTION and spit out the various actions as well - maybe not needed here, but that approach can come in handy in general when mapping data to the CIM.