Splunk Search

Unable to determine why an inline regex using rex works, but not when configured as a field extraction.

Explorer

I am trying to track email sending logs, using information that we adjust in the message_id while sending a message.

sourcetype=postfix_syslog postfix_id=057B41F707AA | rex field=message_id "<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>" | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

During an inline search, those three extracted fields populate the table with integer values, as expected. I then update the field extractions for the sourcetype of postfix_syslog to include the following:

message_id=<o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> 

When I run the subsequent search listed below, those fields in the table are not populated.

sourcetype=postfix_syslog postfix_id=057B41F707AA | table postfix_id message_id email_sender email_identifier email_parameter sourcetype

I am using field discovery on the search, and none of those three fields are included on the discovered fields list.

Any help would be appreciated.

0 Karma
1 Solution

Contributor

Edit:

Sorry, I am mistaken here: I just had to read carefully through the docs:

You can also configure transforms to:

Extract fields from the values of another field (other than _raw) by using the SOURCE_KEY attribute.

In other words: If you would like to to a search-time field extraction on basis of another search-time field do it with a REPORT-Extraction and transforms.conf. Exactly as @somesoni2 describes.


Extending your corresponding props.conf WILL NOT WORK:

[postfix_syslog]
...
EXTRACT-emailfields = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> in message_id 

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

View solution in original post

Contributor

Edit:

Sorry, I am mistaken here: I just had to read carefully through the docs:

You can also configure transforms to:

Extract fields from the values of another field (other than _raw) by using the SOURCE_KEY attribute.

In other words: If you would like to to a search-time field extraction on basis of another search-time field do it with a REPORT-Extraction and transforms.conf. Exactly as @somesoni2 describes.


Extending your corresponding props.conf WILL NOT WORK:

[postfix_syslog]
...
EXTRACT-emailfields = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org> in message_id 

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

View solution in original post

Explorer

I was under the understanding that using the field extractions option in the UI was the equivalent of updating the props.conf with the EXTRACT above. Am I thoroughly off-base here?

0 Karma

Contributor

Conceptually the functionality of the UI has to be a subset of what splunk can accomplish as a whole. This may be a big subset, but I think there will always be things that are not fully implemented in the UI.

Revered Legend

Agreed. Till the time you become proficient in Splunk, you can use UI to do most of your stuff, but once you get a handle of things, you'll find implementing through configuration files is much more simpler/straightforward.

Revered Legend

I believe you would have to create a field transform for this extraction as you're not using _raw field values. See this on how to use field transform to setup this field extraction. After that, once restarted/reloaded, you should be able to use them in your search.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Managefieldtransforms
http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextract...

Explorer

Within the same set of logs there are existing, working, extractions for discovered fields not using _raw.

0 Karma

Revered Legend

Instead of _raw, try to use your field "message_id" as the Source Key. It should be like this for you

props.conf
[Yoursourcetype]
...existing entries...
REPORT-msgfields = getmessagefields

transforms.conf
[getmessagefields]
SOURCE_KEY = message_id
REGEX = <o-(?P<email_sender>\d+).c-(?P<email_identifier>\d+).m-(?P<email_parameter>\d+)@foobar.org>

Explorer

Thank you. I'll take a look.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!