Getting Data In

Help with custom field extraction during indexing

damucka
Contributor

Hello,

I monitor several VMs and the corresponding database logs with the following stanza:[monitor:///usr/sap/.../HDB02/vhsfh*/trace/DB_*/*

The example of the path would be:

/usr/sap/NDJ/HDB02/vhsfhndjdb01/trace/DB_VU7

Now, I would like to extract during indexing from the above path, which is the field "source" additional default fields that are always there, which would be:

SYSTEMDB  (here: NDJ)

vhost (here: vhsfhndjdb01)

DBSID (here: VU7)

The corresponding regex expression, if I would do this in the search is:

| rex field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

 

But how would I do this in the props.conf and transform.conf to make it permanent and always available as the default fields?

I was thinking of the following:

props.conf

[default]

TRUNCATE=0
TRANSFORMS-ExtractKeyFields = fieldextraction
...
 
transforms.conf
[fieldextraction]
REGEX field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
 
but I am not sure about the syntax in the transforms.conf.
Could anyone advice?
Also, what I am obviously trying to do is to set additional default fields based on other default field which is "source". Is it possible at all? Would the "source" already exist and be available at that point to create additional fields out of it?
 
Last, do I need additionally put the fields.conf on my SH?
 
Kind Regards,
Kamil
 
 
Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

As you suspected, the transform is incorrect.  Try this

[fieldextraction]
REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
SOURCE_KEY = MetaData:Source
FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3
---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

As you suspected, the transform is incorrect.  Try this

[fieldextraction]
REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
SOURCE_KEY = MetaData:Source
FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3
---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

damucka
Contributor

Hello,

Thank you. It did not work straight away, but after a bit googling and try-errorying the following worked (interestingly, the REGEX itself had to be changed, I would never come up with this myself, it is also not really documented or at least not so easily understandable for me):

[fieldextraction]
REGEX=^source::\/usr\/sap\/(?P<SYSTEMDB>.+)\/SplunkTest\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)\/.+$
SOURCE_KEY=MetaData:Source
FORMAT=SYSTEMDB::$1 vhost::$2 DBSID::$3
WRITE_META=true

Kind Regards,

Kamil

Tags (1)
0 Karma

soutamo
SplunkTrust
SplunkTrust
Yes you should put fields.conf to sh so it knows that those fields are indexed. Otherwise it don’t, unless you are using list on your with <field>::<value> notation.
R. Ismo
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.