Getting Data In

Help with props and transforms

Champion

Hi,

I have a feed where the fields are separated by brackets (<>). I have a transforms.conf that extracts the fields automatically:

REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

Unfortunately, the fields are all uppercase. I don't see any way to make the fields lowercase, so I've started creating aliases, using FIELD_ALIAS. We need to do this so that they are caught by our ES rules. I also need to do a transforms to map the values appropriately.

Here is a sample field: "Allow

I want to create an alias with the field name to be "action" and a transform that makes the value "allowed". I get the new field, but the transform is not working. Here's what I have configured:

props.conf:

FIELDALIAS-action = ACTION as action

transforms.conf:

[forcepoint_xml]
REGEX = <([^\/][^>]+)>(.*?)<\/[^>]+>
FORMAT = $1::$2
MV_ADD = true

[ACTION]
REGEX = (Allow|Permit)
FORMAT = ACTION::allowed

Any suggestions?

0 Karma

Ultra Champion

How about a sample of the _raw with all your corporate stuff removed. That will help us help you.

0 Karma

Ultra Champion

Couldn't this all be done with one SED command (building on @maciep's suggestion)? I haven't tested but I imagine something with capture groups and forcing lowercase. This essentially combines what everyone else has said.

SED-yaay = s/>(allow|permit)</\L\1/i

The syntax is prob all wrong but the point is that instead of capturing anything you're capturing the specific terms you wanted, using the /i at the end to do case irrelevant. Using the \L that @woodcock introduced to force case on output.

0 Karma

Esteemed Legend

You can convert everything in the raw event to lower-case like this:

SEDCMD-ToLowerCase = s/\(.*\)/\L\1/
0 Karma

Ultra Champion

You might want to make that a bit more specific to only make the keys lowercase, not the values.

0 Karma

Champion

I thought about that, but what's the load when doing that? It's not a super busy feed, but not small either.

0 Karma

Esteemed Legend

There's only one way to know: YOLO!

0 Karma

Ultra Champion

Not entirely sure why your current transforms is not working (except that you are not showing the respective props.conf line) but this is typically resolved with calculated fields or lookups, rather than transforms.

For example, in props.conf (this also gets rid of the need for a FIELDALIAS):

EVAL-action = case(ACTION="Allow","allowed",ACTION="Permit","allowed")

When the number of options grows, a lookup might be a more suitable approach.

PS: any specific reason you're using a custom regex, rather than setting KV_MODE = xml in props.conf?

0 Karma

Champion

It's not proper xml - just bracket seperated fields, so the KV_MODE doesnt' work.

0 Karma

Champion

you might be able to just use eval instead of a field alias. Maybe like this...obviously you can additional allowed/blocked logic:

EVAL-action = if(match(ACTION,"(?i)allow|permit"),"allowed","blocked")

You could also create an automatic lookup to lookup ACTION and spit out the various actions as well - maybe not needed here, but that approach can come in handy in general when mapping data to the CIM.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!