Getting Data In

Help with custom field extraction during indexing

damucka
Builder

Hello,

I monitor several VMs and the corresponding database logs with the following stanza:[monitor:///usr/sap/.../HDB02/vhsfh*/trace/DB_*/*

The example of the path would be:

/usr/sap/NDJ/HDB02/vhsfhndjdb01/trace/DB_VU7

Now, I would like to extract during indexing from the above path, which is the field "source" additional default fields that are always there, which would be:

SYSTEMDB  (here: NDJ)

vhost (here: vhsfhndjdb01)

DBSID (here: VU7)

The corresponding regex expression, if I would do this in the search is:

| rex field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

 

But how would I do this in the props.conf and transform.conf to make it permanent and always available as the default fields?

I was thinking of the following:

props.conf

[default]

TRUNCATE=0
TRANSFORMS-ExtractKeyFields = fieldextraction
...
 
transforms.conf
[fieldextraction]
REGEX field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
 
but I am not sure about the syntax in the transforms.conf.
Could anyone advice?
Also, what I am obviously trying to do is to set additional default fields based on other default field which is "source". Is it possible at all? Would the "source" already exist and be available at that point to create additional fields out of it?
 
Last, do I need additionally put the fields.conf on my SH?
 
Kind Regards,
Kamil
 
 
Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

As you suspected, the transform is incorrect.  Try this

[fieldextraction]
REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
SOURCE_KEY = MetaData:Source
FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

As you suspected, the transform is incorrect.  Try this

[fieldextraction]
REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
SOURCE_KEY = MetaData:Source
FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3
---
If this reply helps you, Karma would be appreciated.
0 Karma

damucka
Builder

Hello,

Thank you. It did not work straight away, but after a bit googling and try-errorying the following worked (interestingly, the REGEX itself had to be changed, I would never come up with this myself, it is also not really documented or at least not so easily understandable for me):

[fieldextraction]
REGEX=^source::\/usr\/sap\/(?P<SYSTEMDB>.+)\/SplunkTest\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)\/.+$
SOURCE_KEY=MetaData:Source
FORMAT=SYSTEMDB::$1 vhost::$2 DBSID::$3
WRITE_META=true

Kind Regards,

Kamil

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
Yes you should put fields.conf to sh so it knows that those fields are indexed. Otherwise it don’t, unless you are using list on your with <field>::<value> notation.
R. Ismo
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...