topic Re: Field extraction not working on distributed search in Splunk Search

Field extraction not working on distributed search

bdstark — Wed, 14 Aug 2013 05:07:24 GMT

I am trying to create a field extraction using the manger to extract the equivalent of:

sourcetype=jsonLogs | rex field=message "^'(?<s>[^']*)' request received"

which extracts requesttype properly (as s - Markdown is removing anything between <> with more than one character). Manager creates the following (ignoring the naming difference between transactiontype and requesttype):

props.conf:
[jsonLogs]
REPORT-requesttype = requesttype

transforms.conf:
[requesttype]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = ^'(?<transactiontype>[^']*)' request received
SOURCE_KEY = message

however the requesttype field is never extracted and does not show up in the available fields dialog with the following search:

sourcetype=jsonLogs

Including the field in the search does not return any results:

sourcetype=jsonLogs requesttype=*

The files are in the search app on the search server, and do not exist on the indexer, which looks to be correct from the documentation at http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Whatisdistributedsearch. The field extraction only occurs on a limited number of results (~65 of 22,000) in the initial search. I can't figure out what I am doing wrong here after working on this for several hours. Is there anything I should be looking at?

Re: Field extraction not working on distributed search

kristian_kolb — Wed, 14 Aug 2013 06:54:51 GMT

In your configuration requesttype is not the name of a field. It is the name of a transform stanza to be applied to data with the jsonLogs sourcetype. The name of the field is what you specify between the angle brackets in your regex statement (in your case that is transactiontype).

As you have correctly noted, this configuration should be placed on the Search Head.

Hope this helps,

Kristian

Re: Field extraction not working on distributed search

Ayn — Wed, 14 Aug 2013 07:53:42 GMT

Moreover, the (?<fieldname>...) format is only for EXTRACT statements, not extractions in transforms.conf. In transforms.conf, you create your matching group without giving it a name - this is done separately in the FORMAT definition. Like this:

REGEX = ^'([^']*)' request received
FORMAT = transactiontype::$1

Re: Field extraction not working on distributed search

bdstark — Wed, 14 Aug 2013 15:44:25 GMT

Missed the transactiontype extract, although that field was also not being extracted (I renamed the field halfway though this thinking transactiontype may be reserved). I have updated as follows based on Ayn's comments:

REGEX = ^'([^']*)' request received
FORMAT = requesttype::$1

The field is still not being extracted with the above search.

Re: Field extraction not working on distributed search

bdstark — Wed, 14 Aug 2013 15:46:08 GMT

I have updated the transforms.prop file, based on Ayn's comments below to:

[requesttype]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = ^'([^']*)' request received
SOURCE_KEY = message
FORMAT = requesttype::$1

The field still is not being extracted.