Problem extracting field from a field

[chengka]

Splunk 6.2
I used the Field Extractor app to extract a field from an previous field. The resulting extraction tested perfectly from with the app. Here is an excerpt from the sourcetype stanza of props.conf.

EXTRACT-ErrorMessageText = (?ims)^\w+.+AMQ\d\d\d\d:(?P.+)EXPLANATION
EXTRACT-ErrorMessageChannel = [cC]hannel[\s']+(?P[\w\d.]+) in ErrorMessageText

The problem is the field ErrorMessageChannel field never appears in the field list for an event. The 6.2 field extractor does not even list it as an Existing Field (ErrorMessageText is listed). The new field IS listed in the Settings Field Extractions.

Example event:

12/10/2014 03:54:45 PM - Process(5136.740878) User(mqm) Program(amqrmppa)
Host(hostname) Installation(Installation_2)
VRMF(7.5.0.3) QMgr(abcdef)
AMQ9999: Channel 'ABCD.EFG' to host 'xxx-xxx (10.1.1.1)' ended abnormally. EXPLANATION: The channel program running under process ID 5136 for channel 'ABCD.EFG' ended abnormally. The host name is 'xxx-xxx (10.1.1.1)'; in some cases the host name cannot be determined and so is shown as '????'. ACTION: Look at previous error messages for the channel program in the error logs to determine the cause of the failure. Note that this message can be excluded completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage" attributes under the "QMErrorLog" stanza in qm.ini. Further information can be found in the System Administration Guide.

The ErrorMessageText works properly and resolves to
"AMQ9999: Channel 'ABCD.EFG' to host 'xxx-xxx (10.1.1.1)'
ended abnormally."

The regex for ErrorMessageChannel works fine with the Field Extractor app as well. For this event the ErrorMessageText would be "ABCD.EFG"

What am I missing here?

[lguinn2 · Accepted solution]

I think this is your problem: when you specify the EXTRACT, Splunk uses the class that you provide to sequence the order of evaluation. This is particularly important when you consider that Splunk merges all the props.conf files into one run-time specification.

Since ErrorMessageChannel precedes ErrorMessageText, it is performed first and extracts nothing.

Do this instead:

 EXTRACT-e1 = (?ims)^\w+.+AMQ\d\d\d\d:(?P<ErrorMessageText>.+)EXPLANATION
 EXTRACT-e2 = [cC]hannel[\s']+(?P[\w\d.]+) in ErrorMessageTex

and I think it will work. BTW, the class name is not at all related to the field name.

[chengka]

Thank you, changing the class name did the trick.

[somesoni2]

Try adding "| extract reload=t" at the end of your search. Also, see if restarting your search head is an option and try doing that.

[chengka]

I can see the field when I add "| extract reload=t". When I remove it, the field disappears. Also I adding ErrorMessageChannel=* or any of the Top 10 values returns nothing.

[lguinn2]

If it was actually a reload problem, then you should only need to run this (by itself, not at the end of a search) one time

| reload extract=t

Can you show us the actual search, not just the props.conf entries?

[jhupka]

Can you provide a sample event? If it is listed in the Field Extractions settings that means Splunk knows it is defined...but if you're not seeing it within the Existing Field list on the left of your search results there might be some other issue in your RegEx that isn't working right so nothing ever gets extracted.

As a side note, it looks like maybe the less-than and greater-than portions of your field extractions got left out by the website - you might want to try editing it and markup your EXTRACT lines as blocks of code.