Splunk Search

Field extraction not working on distributed search

bdstark
New Member

I am trying to create a field extraction using the manger to extract the equivalent of:

sourcetype=jsonLogs | rex field=message "^'(?<s>[^']*)' request received"

which extracts requesttype properly (as s - Markdown is removing anything between <> with more than one character). Manager creates the following (ignoring the naming difference between transactiontype and requesttype):

props.conf:
[jsonLogs]
REPORT-requesttype = requesttype

transforms.conf:
[requesttype]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = ^'(?<transactiontype>[^']*)' request received
SOURCE_KEY = message

however the requesttype field is never extracted and does not show up in the available fields dialog with the following search:

sourcetype=jsonLogs

Including the field in the search does not return any results:

sourcetype=jsonLogs requesttype=*

The files are in the search app on the search server, and do not exist on the indexer, which looks to be correct from the documentation at http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Whatisdistributedsearch. The field extraction only occurs on a limited number of results (~65 of 22,000) in the initial search. I can't figure out what I am doing wrong here after working on this for several hours. Is there anything I should be looking at?

0 Karma

bdstark
New Member

I have updated the transforms.prop file, based on Ayn's comments below to:

[requesttype]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = ^'([^']*)' request received
SOURCE_KEY = message
FORMAT = requesttype::$1

The field still is not being extracted.

0 Karma

kristian_kolb
Ultra Champion

In your configuration requesttype is not the name of a field. It is the name of a transform stanza to be applied to data with the jsonLogs sourcetype. The name of the field is what you specify between the angle brackets in your regex statement (in your case that is transactiontype).

As you have correctly noted, this configuration should be placed on the Search Head.

Hope this helps,

Kristian

0 Karma

bdstark
New Member

Missed the transactiontype extract, although that field was also not being extracted (I renamed the field halfway though this thinking transactiontype may be reserved). I have updated as follows based on Ayn's comments:

REGEX = ^'([^']*)' request received
FORMAT = requesttype::$1

The field is still not being extracted with the above search.

0 Karma

Ayn
Legend

Moreover, the (?<fieldname>...) format is only for EXTRACT statements, not extractions in transforms.conf. In transforms.conf, you create your matching group without giving it a name - this is done separately in the FORMAT definition. Like this:

REGEX = ^'([^']*)' request received
FORMAT = transactiontype::$1
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!