topic Re: Optimize rex command in Splunk Search

Optimize rex command

clementros — Wed, 30 Sep 2020 03:24:27 GMT

Hi all,

I want to extract fields form log events.

I have two errors patterns :

EDICPP 4-1-1-0 exception: Mandatory element is missing.  Error at <src/CBRSegmentDecoders.cpp:21405 Error in segment CSC 06 1 1A TKA in element #2. 
    ------  Error at <src/CBRMessageDecoders.cpp:121 Error in group Group id: _294728_G_CBR in element #3. 
    ------  Decoding error in message AIRRQT 15 2 1A PNR in element #4 at character 240 of the buffer, using charset B  Converted segments: ORG - BLK - AMD - AVC - BIN -  Cannot convert CSC from here==>+9998WSSU'Last segment correctly processed: CSC+7906:/++9998WSSU'

and

Can't find the UNH segment.

What i want is to create a field named grammar_error_reason that contain exactly "Can not convert FOO" or "Can't find the FOO segment".

Here is my rex command :

(Converted segments:(\s\w{3}+\s-)+\s|)(?<grammar_error_reason>.+) (from here|segment\.)

Can you help me please.

Kind regards,
Clement

Re: Optimize rex command

richgalloway — Wed, 18 Dec 2019 14:00:07 GMT

This regex may work for you. If it doesn't, consider using two rex commands - one for each error pattern,

... | rex "(?J)(Converted segments:(\s\w{3}+\s-)+\s+(?<grammar_error_reason>.+) from here|(?<grammar_error_reason>Can't find the \w+ segment\.))"

Re: Optimize rex command

clementros — Wed, 18 Dec 2019 14:25:12 GMT

Hi @richgalloway,

Unfortunatly your rex command do not work cause it exceeded configured match_limit.

If i use two rex command to extract info i think i will have a problem. The regex to extract the second pattern "Can't find the UNH segment." also match with the other pattern.

How can i say if you match with the first do not try the second ?

Re: Optimize rex command

richgalloway — Wed, 18 Dec 2019 14:54:12 GMT

Try adding the max_match=0 option to the rex command.

Two separate rex commands should work since they each search for different strings. We can avoid conflicts using two fields and coalesce.

... | rex "(Converted segments:(\s\w{3}+\s-)+\s+(?<grammar_error_reason1>.+) from here" 
| rex "(?<grammar_error_reason2>Can't find the \w+ segment\.)"
| eval grammar_error_reason = coalesce(grammar_reason1, grammar_error_reason2)

Re: Optimize rex command

clementros — Wed, 18 Dec 2019 15:28:14 GMT

Thank you @richgalloway,

This command works for me :

index="dbtkbkmalformed" CDCATEGORY=GRAMMAR_ERROR 
| rex "Converted segments:(\s\w{3}+\s-)+\s+(?<grammar_error_reason1>[\w\s]+) from here" 
| rex "(?<grammar_error_reason2>Can't find the \w+ segment\.)" 
| eval grammar_error_reason = coalesce(grammar_reason1, grammar_error_reason2)

Re: Optimize rex command

woodcock — Wed, 18 Dec 2019 20:35:54 GMT

Be sure to do something with your duplicate question:
https://answers.splunk.com/answers/790422/optimize-rex-command.html