Help with custom field extraction during indexing

[damucka]

Hello,

I monitor several VMs and the corresponding database logs with the following stanza:[monitor:///usr/sap/.../HDB02/vhsfh*/trace/DB_*/*

The example of the path would be:

/usr/sap/NDJ/HDB02/vhsfhndjdb01/trace/DB_VU7

Now, I would like to extract during indexing from the above path, which is the field "source" additional default fields that are always there, which would be:

SYSTEMDB  (here: NDJ)

vhost (here: vhsfhndjdb01)

DBSID (here: VU7)

The corresponding regex expression, if I would do this in the search is:

| rex field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

But how would I do this in the props.conf and transform.conf to make it permanent and always available as the default fields?

I was thinking of the following:

props.conf

[default]

TRUNCATE=0

TRANSFORMS-ExtractKeyFields = fieldextraction

...

transforms.conf

[fieldextraction]

REGEX field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

but I am not sure about the syntax in the transforms.conf.

Could anyone advice?

Also, what I am obviously trying to do is to set additional default fields based on other default field which is "source". Is it possible at all? Would the "source" already exist and be available at that point to create additional fields out of it?

Last, do I need additionally put the fields.conf on my SH?

Kind Regards,

Kamil

[richgalloway · Accepted solution]

As you suspected, the transform is incorrect.  Try this

[fieldextraction]
REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"
SOURCE_KEY = MetaData:Source
FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3

[damucka]

Hello,

Thank you. It did not work straight away, but after a bit googling and try-errorying the following worked (interestingly, the REGEX itself had to be changed, I would never come up with this myself, it is also not really documented or at least not so easily understandable for me):

[fieldextraction]
REGEX=^source::\/usr\/sap\/(?P<SYSTEMDB>.+)\/SplunkTest\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)\/.+$
SOURCE_KEY=MetaData:Source
FORMAT=SYSTEMDB::$1 vhost::$2 DBSID::$3
WRITE_META=true

Kind Regards,

Kamil

[isoutamo]

Yes you should put fields.conf to sh so it knows that those fields are indexed. Otherwise it don’t, unless you are using list on your with <field>::<value> notation.
R. Ismo