topic Re: Field extraction during indexing does not work in Getting Data In

Field extraction during indexing does not work

damucka — Mon, 12 Oct 2020 15:15:51 GMT

Hello,

I have following entry in my transforms.conf:

[dtimes] REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0$timeofday$:(?P<t0>.*?);dt1$us$:(?P<dt1>.*?);dt2$us$:(?P<dt2>.*?);dt3$us$:(?P<dt t4>.*?);total$us$:(?P<total>.*?)$ SOURCE_KEY=_raw FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7 WRITE_META=true

which I would expect extract the corresponding fields out of the events like the one below:

[12/Oct/2020:03:56:39 +0000] 10.1.6.58 100/CB9980000122 100/CB9980000122 042457C44BD441A36E673571F0C7D1AF - "GET /sap/bc/ui5_ui5/sap/fin_lib/~D0C2FE335CFD0450BE39DFA0391E81C6~5/error/Error.js HTTP/2" 200 1081 - 2ms my303891.s4hana.ondemand.com NII vhsfhniici_NII_00 "-"TLSv1.2 t0(timeofday):1602474999.837288;dt1(us):501;dt2(us):32;dt3(us):1257;dt4(us):34;total(us):1824

As per regex101 it works fine, also the SPL search with the above rex field=_raw works fine. Unfortunately when placing it in the transforms.conf it does not.

There are also the matching entries in the props.conf:

[webdispatcher] TRANSFORMS-ExtractKeyFields = dtimes TRANSFORMS-ExtractKeyFields = passportID

and fields.conf:

[SYSTEMDB] INDEXED = True INDEXED_VALUE = False [vhost] INDEXED = True INDEXED_VALUE = False [DBSID] INDEXED = True INDEXED_VALUE = False # ############### Extract the performance KPIs from the Webdispatcher trace [passportID] INDEXED = True INDEXED_VALUE = False [request] INDEXED = True INDEXED_VALUE = False [status] INDEXED = True INDEXED_VALUE = False [t0] INDEXED = True INDEXED_VALUE = False [dt1] INDEXED = True INDEXED_VALUE = False [dt2] INDEXED = True INDEXED_VALUE = False [dt3] INDEXED = True INDEXED_VALUE = False [dt4] INDEXED = True INDEXED_VALUE = False [total] INDEXED = True INDEXED_VALUE = False #******************************

Can anyone help?

The second regex there (passportID), which is sligtly easier, works fine ...

Kind Regards,

Kamil

Re: Field extraction during indexing does not work

damucka — Mon, 12 Oct 2020 17:11:17 GMT

... just noticed that the transforms.conf entry got messed up when pasting. The correct entry looks as follows:

[dtimes] REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0$timeofday$:(?P<t0>.*?);dt1$us$:(?P<dt1>.*?);dt2$us$:(?P<dt2>.*?);dt3$us$:(?P<dt3>.*?);dt4$us$:(?P<dt4>.*?);total$us$:(?P<total>.*?)$ SOURCE_KEY=_raw FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7 WRITE_META=true

Re: Field extraction during indexing does not work

richgalloway — Mon, 12 Oct 2020 17:15:38 GMT

The FORMAT line references 7 capture groups, but the REGEX line defines only 6.

Re: Field extraction during indexing does not work

damucka — Tue, 13 Oct 2020 06:57:57 GMT

Hi,

That was copy-paste mistake, please see my update.

Both have 7 groups.

Kind Regards,

Kamil

Re: Field extraction during indexing does not work

richgalloway — Tue, 13 Oct 2020 13:40:44 GMT

It's not necessary for the regular expression to describe every character in the event. Try this regex. It uses fewer steps so at the very least it should perform faster.

\.s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0$timeofday$:(?P<t0>.*?);dt1$us$:(?P<dt1>.*?);dt2$us$:(?P<dt2>.*?);dt3$us$:(?P<dt3>.*?);dt4$us$:(?P<dt4>.*?);total$us$:(?P<total>.*)

Re: Field extraction during indexing does not work

damucka — Tue, 13 Oct 2020 15:03:55 GMT

The issue was in the props.conf.

It was:

[webdispatcher] TRANSFORMS-ExtractKeyFields = dtimes TRANSFORMS-ExtractKeyFields = passportID

and it should be:

[webdispatcher] TRANSFORMS-ExtractKeyFields = dtimes, passportID

I did not know about that. The regex itself was fine.

Thank you for your help.

Kind regards,

Kamil