Unable to determine why an inline regex using rex works, but not when configured as a field extraction.

[dfenko]

I am trying to track email sending logs, using information that we adjust in the message_id while sending a message.

sourcetype=postfix_syslog postfix_id=057B41F707AA | rex field=message_id "<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>" | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

During an inline search, those three extracted fields populate the table with integer values, as expected. I then update the field extractions for the sourcetype of postfix_syslog to include the following:

message_id=<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> 

When I run the subsequent search listed below, those fields in the table are not populated.

sourcetype=postfix_syslog postfix_id=057B41F707AA | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

I am using field discovery on the search, and none of those three fields are included on the discovered fields list.

Any help would be appreciated.

[bjoernjensen · Accepted solution]

Edit:

Sorry, I am mistaken here: I just had to read carefully through the docs:

You can also configure transforms to:

Extract fields from the values of another field (other than _raw) by using the SOURCE_KEY attribute.

In other words: If you would like to to a search-time field extraction on basis of another search-time field do it with a REPORT-Extraction and transforms.conf. Exactly as @somesoni2 describes.

---

Extending your corresponding props.conf WILL NOT WORK:

[postfix_syslog]
...
EXTRACT-emailfields = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> in message_id 

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

[dfenko]

I was under the understanding that using the field extractions option in the UI was the equivalent of updating the props.conf with the EXTRACT above. Am I thoroughly off-base here?

[bjoernjensen]

Conceptually the functionality of the UI has to be a subset of what splunk can accomplish as a whole. This may be a big subset, but I think there will always be things that are not fully implemented in the UI.

[somesoni2]

Agreed. Till the time you become proficient in Splunk, you can use UI to do most of your stuff, but once you get a handle of things, you'll find implementing through configuration files is much more simpler/straightforward.

[somesoni2]

I believe you would have to create a field transform for this extraction as you're not using _raw field values. See this on how to use field transform to setup this field extraction. After that, once restarted/reloaded, you should be able to use them in your search.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Managefieldtransforms
http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

[dfenko]

Within the same set of logs there are existing, working, extractions for discovered fields not using _raw.

[somesoni2]

Instead of _raw, try to use your field "message_id" as the Source Key. It should be like this for you

props.conf
[Yoursourcetype]
...existing entries...
REPORT-msgfields = getmessagefields

transforms.conf
[getmessagefields]
SOURCE_KEY = message_id
REGEX = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>

[dfenko]

Thank you. I'll take a look.