Field extraction during indexing does not work

[damucka]

Hello,

I have following entry in my transforms.conf:

[dtimes]
REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt
t4>.*?);total\(us\):(?P<total>.*?)$
SOURCE_KEY=_raw
FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7
WRITE_META=true

which I would expect extract the corresponding fields out of the events like the one below:

[12/Oct/2020:03:56:39 +0000] 10.1.6.58 100/CB9980000122 100/CB9980000122 042457C44BD441A36E673571F0C7D1AF - "GET /sap/bc/ui5_ui5/sap/fin_lib/~D0C2FE335CFD0450BE39DFA0391E81C6~5/error/Error.js HTTP/2" 200 1081 - 2ms my303891.s4hana.ondemand.com NII vhsfhniici_NII_00 "-"TLSv1.2 t0(timeofday):1602474999.837288;dt1(us):501;dt2(us):32;dt3(us):1257;dt4(us):34;total(us):1824

As per regex101 it works fine, also the SPL search with the above rex field=_raw works fine. Unfortunately when placing it in the transforms.conf it does not.

There are also the matching entries in the props.conf:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes
TRANSFORMS-ExtractKeyFields = passportID

and fields.conf:

[SYSTEMDB]
INDEXED = True
INDEXED_VALUE = False

[vhost]
INDEXED = True
INDEXED_VALUE = False

[DBSID]
INDEXED = True
INDEXED_VALUE = False

# ############### Extract the performance KPIs from the Webdispatcher trace
[passportID]
INDEXED = True
INDEXED_VALUE = False

[request]
INDEXED = True
INDEXED_VALUE = False

[status]
INDEXED = True
INDEXED_VALUE = False

[t0]
INDEXED = True
INDEXED_VALUE = False

[dt1]
INDEXED = True
INDEXED_VALUE = False

[dt2]
INDEXED = True
INDEXED_VALUE = False

[dt3]
INDEXED = True
INDEXED_VALUE = False

[dt4]
INDEXED = True
INDEXED_VALUE = False

[total]
INDEXED = True
INDEXED_VALUE = False
#******************************

Can anyone help?

The second regex there (passportID), which is sligtly easier, works fine ...

Kind Regards,

Kamil

[richgalloway · Accepted solution]

It's not necessary for the regular expression to describe every character in the event.  Try this regex.  It uses fewer steps so at the very least it should perform faster.

\.s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt3>.*?);dt4\(us\):(?P<dt4>.*?);total\(us\):(?P<total>.*)

[damucka]

The issue was in the props.conf.

It was:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes
TRANSFORMS-ExtractKeyFields = passportID

and it should be:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes, passportID

I did not know about that. The regex itself was fine.

Thank you for your help.

Kind regards,

Kamil

[richgalloway]

The FORMAT line references 7 capture groups, but the REGEX line defines only 6.

[damucka]

Hi,

That was copy-paste mistake, please see my update.

Both have 7 groups.

Kind Regards,

Kamil

[damucka]

... just noticed that the transforms.conf entry got messed up when pasting. The correct entry looks as follows:

[dtimes]
REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt3>.*?);dt4\(us\):(?P<dt4>.*?);total\(us\):(?P<total>.*?)$
SOURCE_KEY=_raw
FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7
WRITE_META=true