Splunk Search

Unable to determine why an inline regex using rex works, but not when configured as a field extraction.

dfenko
Explorer

I am trying to track email sending logs, using information that we adjust in the message_id while sending a message.

sourcetype=postfix_syslog postfix_id=057B41F707AA | rex field=message_id "<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>" | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

During an inline search, those three extracted fields populate the table with integer values, as expected. I then update the field extractions for the sourcetype of postfix_syslog to include the following:

message_id=<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> 

When I run the subsequent search listed below, those fields in the table are not populated.

sourcetype=postfix_syslog postfix_id=057B41F707AA | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

I am using field discovery on the search, and none of those three fields are included on the discovered fields list.

Any help would be appreciated.

0 Karma
1 Solution

bjoernjensen
Contributor

Edit:

Sorry, I am mistaken here: I just had to read carefully through the docs:

You can also configure transforms to:

Extract fields from the values of another field (other than _raw) by using the SOURCE_KEY attribute.

In other words: If you would like to to a search-time field extraction on basis of another search-time field do it with a REPORT-Extraction and transforms.conf. Exactly as @somesoni2 describes.


Extending your corresponding props.conf WILL NOT WORK:

[postfix_syslog]
...
EXTRACT-emailfields = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> in message_id 

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

View solution in original post

bjoernjensen
Contributor

Edit:

Sorry, I am mistaken here: I just had to read carefully through the docs:

You can also configure transforms to:

Extract fields from the values of another field (other than _raw) by using the SOURCE_KEY attribute.

In other words: If you would like to to a search-time field extraction on basis of another search-time field do it with a REPORT-Extraction and transforms.conf. Exactly as @somesoni2 describes.


Extending your corresponding props.conf WILL NOT WORK:

[postfix_syslog]
...
EXTRACT-emailfields = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> in message_id 

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

dfenko
Explorer

I was under the understanding that using the field extractions option in the UI was the equivalent of updating the props.conf with the EXTRACT above. Am I thoroughly off-base here?

0 Karma

bjoernjensen
Contributor

Conceptually the functionality of the UI has to be a subset of what splunk can accomplish as a whole. This may be a big subset, but I think there will always be things that are not fully implemented in the UI.

somesoni2
Revered Legend

Agreed. Till the time you become proficient in Splunk, you can use UI to do most of your stuff, but once you get a handle of things, you'll find implementing through configuration files is much more simpler/straightforward.

somesoni2
Revered Legend

I believe you would have to create a field transform for this extraction as you're not using _raw field values. See this on how to use field transform to setup this field extraction. After that, once restarted/reloaded, you should be able to use them in your search.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Managefieldtransforms
http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

dfenko
Explorer

Within the same set of logs there are existing, working, extractions for discovered fields not using _raw.

0 Karma

somesoni2
Revered Legend

Instead of _raw, try to use your field "message_id" as the Source Key. It should be like this for you

props.conf
[Yoursourcetype]
...existing entries...
REPORT-msgfields = getmessagefields

transforms.conf
[getmessagefields]
SOURCE_KEY = message_id
REGEX = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>

dfenko
Explorer

Thank you. I'll take a look.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...