topic Re: Regexp for transform.conf doesnt work in Getting Data In

Regexp for transform.conf doesnt work

evkuzin — Wed, 15 Nov 2017 14:22:47 GMT

Splunk receive a log like this:

Nov 15 13:02:10 172.20.20.3 test WARNING 1 "Invalid path" 178.217.60.3 0 10.18.7.98 2040 5 "bla bla bla" sampled 1 0 N/A low drop FFFFFFFF-FFFF-FFFF-000E-000059C98546

And vendor said that each field is separated by a single space character. Fields that may contain spaces are printed between double quotes. So for each field i use configuration like this:

props.conf:

[test_source]
REPORT-device_ip=device_ip
REPORT-attack_name=attack_name

transforms.conf

[device_ip]
REGEX = (".*?"|\S+)
FORMAT = device_ip::$4
MV_ADD = true

[attack_name]
REGEX = (".*?"|\S+)
FORMAT = attack_name::$8
MV_ADD = true

I've already test regexp with https://regex101.com/ and it should just split fields in accordance to vendors documentation, but it doesn't work.

all files I put in /opt/splunk/etc/system/local/

Re: Regexp for transform.conf doesnt work

richgalloway — Tue, 29 Sep 2020 16:44:06 GMT

The second IP address ("defence_attack_name") is group #9, not 8.
When you say "it doesn't work", what exactly do you mean? What results are you getting?

Re: Regexp for transform.conf doesnt work

evkuzin — Tue, 29 Sep 2020 16:44:09 GMT

Actually I'm getting nothing 🙂
I mean that nothing extracted at all except field vendor_product which defined as EVAL-vendor_product = "Radware" in props.conf
You can see it on screenshot https://ibb.co/d6qhoR
How could I debug fields extraction?

Re: Regexp for transform.conf doesnt work

joebisesi — Wed, 28 Nov 2018 15:13:28 GMT

Have you tried setting the MV_Add to its default value? Which is false.

Re: Regexp for transform.conf doesnt work

FrankVl — Wed, 28 Nov 2018 15:31:26 GMT

Your regex contains just 1 capturing group, so how can you refer to $4 and $8?

Yes, this regex matches all the bits separated by spaces, but this is not how you extract fields from an event like this.

Approaches that would work:

REGEX = (".*?"|\S+)\s(".*?"|\S+)\s(".*?"|\S+)\s etc.
FORMAT = field1::$1 field2::$2 field3::$3 etc.

Which uses a regex that matches the event as a whole and extracts each field into a capture group and then assigns those capture groups to the relevant fieldname.

Or:

REGEX = (?:(?:".*?"|\S+)\s){10}(".*?"|\S+)
FORMAT = ip::$1

Which skips over X (10 in this example) fields and then extracts a single piece to assign to a specific field.