Splunk Search
Highlighted

Receiving message "Field extractor name=extract-doublecolon-transform is unusually slow" How to optimize the regex for my field extraction?

Contributor

We have events in below format..

[2014-11-17 05:00:00,876] [INFO] [EventTimestamp::2014-11-17T05:00:00.876-06:00|ReferenceID::SomeID|ServiceName::Some.Services|OperationName::<null>|Direction::REQUEST|Server.Port::prod_domain.server1:1001|<xml>...some_big_xml_here...</xml>]

We applied below props/transforms to extract fields, with field_name on left side of :: and right side the value
(something similar to what splunk does by default with = sign in logs)

[extract-doublecolon-transform]
REGEX=([^\s\:]+)\:\:([^\|]+)\|
FORMAT=$1::$2

This regex works fine, however at times I receive below message.
Field extractor name=extract-doublecolon-transform is unusually slow
How do I best optimize the above regex for the sample event given above.

0 Karma
Highlighted

Re: Receiving message "Field extractor name=extract-doublecolon-transform is unusually slow" How to optimize the regex for my field extraction?

SplunkTrust
SplunkTrust

Hi Venkat_16,

The solutions are :
- improve the regexes/field extractions ( like this ([^\|\[]+)\:\:([^\|]+) ? )
- or change the warning threshold for key values extraction

edit $SPLUNKHOME/etc/system/local/limits.conf, and change maxextractor_time value
see http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Limitsconf for more details

[kv]
max_extractor_time = <integer>
* Maximum amount of CPU time, in milliseconds, that a key-value pair extractor will be allowed to 
* take before warning. If the extractor exceeds this execution time on any event a warning will be issued
* Defaults to 1000

avg_extractor_time = <integer>
* Maximum amount of CPU time, in milliseconds, that the average (over search results) execution time of 
* a key-value pair extractor will be allowed to take before warning. Once the average becomes larger 
* than this amount of time a warning will be issued
* Defaults to 500

hope this helps to sort things ...

cheers, MuS

0 Karma