topic Re: Help with custom field extraction during indexing in Getting Data In

Help with custom field extraction during indexing

damucka — Fri, 25 Sep 2020 13:38:47 GMT

Hello,

I monitor several VMs and the corresponding database logs with the following stanza:[monitor:///usr/sap/.../HDB02/vhsfh*/trace/DB_*/*

The example of the path would be:

/usr/sap/NDJ/HDB02/vhsfhndjdb01/trace/DB_VU7

Now, I would like to extract during indexing from the above path, which is the field "source" additional default fields that are always there, which would be:

SYSTEMDB (here: NDJ)

vhost (here: vhsfhndjdb01)

DBSID (here: VU7)

The corresponding regex expression, if I would do this in the search is:

| rex field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

But how would I do this in the props.conf and transform.conf to make it permanent and always available as the default fields?

I was thinking of the following:

props.conf

[default]

TRUNCATE=0

TRANSFORMS-ExtractKeyFields = fieldextraction

...

transforms.conf

[fieldextraction]

REGEX field=source "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$"

but I am not sure about the syntax in the transforms.conf.

Could anyone advice?

Also, what I am obviously trying to do is to set additional default fields based on other default field which is "source". Is it possible at all? Would the "source" already exist and be available at that point to create additional fields out of it?

Last, do I need additionally put the fields.conf on my SH?

Kind Regards,

Kamil

Re: Help with custom field extraction during indexing

richgalloway — Fri, 25 Sep 2020 14:33:52 GMT

As you suspected, the transform is incorrect. Try this

[fieldextraction] REGEX = "\/usr\/sap\/(?P<SYSTEMDB>.+)\/HDB02\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)$" SOURCE_KEY = MetaData:Source FORMAT = SYSTEMDB::$1 vhost::$2 DBSID::$3

Re: Help with custom field extraction during indexing

isoutamo — Fri, 25 Sep 2020 15:38:19 GMT

Yes you should put fields.conf to sh so it knows that those fields are indexed. Otherwise it don’t, unless you are using list on your with <field>::<value> notation.
R. Ismo

Re: Help with custom field extraction during indexing

damucka — Tue, 29 Sep 2020 13:30:53 GMT

Hello,

Thank you. It did not work straight away, but after a bit googling and try-errorying the following worked (interestingly, the REGEX itself had to be changed, I would never come up with this myself, it is also not really documented or at least not so easily understandable for me):

[fieldextraction]
REGEX=^source::\/usr\/sap\/(?P<SYSTEMDB>.+)\/SplunkTest\/(?P<vhost>.+)\/trace\/DB_(?P<DBSID>.+)\/.+$
SOURCE_KEY=MetaData:Source
FORMAT=SYSTEMDB::$1 vhost::$2 DBSID::$3
WRITE_META=true

Kind Regards,

Kamil