Why isn't this field transform working?

[dpanych]

Hi everyone, I'm having trouble applying the following fields transformation — it's not "parsing" during search time. The regex works fine, it's parsing VALUE, so for example, CODE=22344, but nothing seems to go on during search time. Any help is much appreciated.

Regex:

<([^>]+)>\h+([^<]+)(?:\h+|$)

Sample data:

Jan 22 09:00:00 10.10.0.190 MCS:BS::REPORT::RUN: <Code> 22344 <Type> AUDIT <Severity> PROCESS <Category> SECURITY <User> root <HwSource> MCGUI <Summary> Report ran <report> /Usage Intelligence - Front End Capacity 

Jan 22 09:00:00 10.10.0.190 MCS:BS::REPORT::RUN: <Code> 22344 <Type> AUDIT <Severity> PROCESS <Category> SECURITY <User> root <HwSource> MCGUI <Summary> Report ran <report> /Usage Intelligence - Backend Capacity

Jan 22 08:51:51 10.10.0.190 MCS:AvmgrLoginModule::USER::LOGOFF: <Code> 22343 <Type> AUDIT <Severity> USER <Category> SECURITY <User> MCUser <HwSource> ss-1r <Summary> User logoff successful <action> logout

[harsmarvania57 · Accepted solution]

Hi,

If you want extract field names and values both from raw data at search time then it is possible using props and transforms.

To accomplish using Splunk Web

1. Settings -> Fields -> Field Transformations -> New Field Transformation, provide any user friendly name; type regex; copy your regex <([^>]+)>\h+([^<]+)(?:\h+|$) and in format provide $1::$2 and tick Automatically clean field names; Save.
2. Settings -> Fields -> Field Extractions -> New Field Extraction, provide any user friendly name; select source, host or sourcetype based on your requirement and provide corresponding value in named input; type - select Uses transform; in extraction/Transform provide use friendly name which you have given in step 1 (while creating field transformation); Save

Now search your data with base query and it will automatically extract fields action, category, .... with their values based on regex. I have tested above steps in my lab environment with sample data you have provided and it is extracting various fields with values.

[dpanych]

This worked! Thank you very much. I had only the first part done initially.

[cpetterborg]

Search time regex like this (I would assume you are doing it with a rex command) can't extract the field name in this manner. You won't get any fields because you aren't using a named capture group, only unnamed capture groups. If you want this to work, it would have to be done at index time. If you can't do it at index time you need to extract the fields with named capture groups, which is probably going to be a PITA to do with events that are not equivalent to each other. You may have to do something like:

... | rex "\<code\> (?<Code>\d+)" | rex "<\Type\> (?<type>[^\<]+)\s" | rex ...