Getting Data In

Field extraction during indexing does not work

damucka
Builder

Hello,

I have following entry in my transforms.conf:

[dtimes]
REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt
t4>.*?);total\(us\):(?P<total>.*?)$
SOURCE_KEY=_raw
FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7
WRITE_META=true

which I would expect extract the corresponding fields out of the events like the one below:

[12/Oct/2020:03:56:39 +0000] 10.1.6.58 100/CB9980000122 100/CB9980000122 042457C44BD441A36E673571F0C7D1AF - "GET /sap/bc/ui5_ui5/sap/fin_lib/~D0C2FE335CFD0450BE39DFA0391E81C6~5/error/Error.js HTTP/2" 200 1081 - 2ms my303891.s4hana.ondemand.com NII vhsfhniici_NII_00 "-"TLSv1.2 t0(timeofday):1602474999.837288;dt1(us):501;dt2(us):32;dt3(us):1257;dt4(us):34;total(us):1824

 

As per regex101 it works fine, also the SPL search with the above rex field=_raw works fine. Unfortunately when placing it in the transforms.conf it does not.

There are also the matching entries in the props.conf:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes
TRANSFORMS-ExtractKeyFields = passportID

 

and fields.conf:

[SYSTEMDB]
INDEXED = True
INDEXED_VALUE = False

[vhost]
INDEXED = True
INDEXED_VALUE = False

[DBSID]
INDEXED = True
INDEXED_VALUE = False

# ############### Extract the performance KPIs from the Webdispatcher trace
[passportID]
INDEXED = True
INDEXED_VALUE = False

[request]
INDEXED = True
INDEXED_VALUE = False

[status]
INDEXED = True
INDEXED_VALUE = False

[t0]
INDEXED = True
INDEXED_VALUE = False

[dt1]
INDEXED = True
INDEXED_VALUE = False

[dt2]
INDEXED = True
INDEXED_VALUE = False

[dt3]
INDEXED = True
INDEXED_VALUE = False

[dt4]
INDEXED = True
INDEXED_VALUE = False

[total]
INDEXED = True
INDEXED_VALUE = False
#******************************

 

Can anyone help?

The second regex there (passportID), which is sligtly easier, works fine ...

Kind Regards,

Kamil

 

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

It's not necessary for the regular expression to describe every character in the event.  Try this regex.  It uses fewer steps so at the very least it should perform faster.

\.s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt3>.*?);dt4\(us\):(?P<dt4>.*?);total\(us\):(?P<total>.*)

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's not necessary for the regular expression to describe every character in the event.  Try this regex.  It uses fewer steps so at the very least it should perform faster.

\.s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt3>.*?);dt4\(us\):(?P<dt4>.*?);total\(us\):(?P<total>.*)

 

---
If this reply helps you, Karma would be appreciated.
0 Karma

damucka
Builder

The issue was in the props.conf.

It was:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes
TRANSFORMS-ExtractKeyFields = passportID

 

and it should be:

[webdispatcher]
TRANSFORMS-ExtractKeyFields = dtimes, passportID

 

I did not know about that. The regex itself was fine.

Thank you for your help.

Kind regards,

Kamil

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The FORMAT line references 7 capture groups, but the REGEX line defines only 6.

---
If this reply helps you, Karma would be appreciated.
0 Karma

damucka
Builder

Hi,

That was copy-paste mistake, please see my update.

Both have 7 groups.

Kind Regards,

Kamil

0 Karma

damucka
Builder

... just noticed that the transforms.conf entry got messed up when pasting. The correct entry looks as follows:

[dtimes]
REGEX = ^.+s4hana\.ondemand\.com (?P<DBSID>.{3}).+t0\(timeofday\):(?P<t0>.*?);dt1\(us\):(?P<dt1>.*?);dt2\(us\):(?P<dt2>.*?);dt3\(us\):(?P<dt3>.*?);dt4\(us\):(?P<dt4>.*?);total\(us\):(?P<total>.*?)$
SOURCE_KEY=_raw
FORMAT = DBSID::$1 t0::$2 dt1::$3 dt2::$4 dt3::$5 dt4::$6 total::$7
WRITE_META=true
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...