Splunk Search

Rex has exceeded configured match_limit

clementros
Path Finder

I am trying to extract about 4 fields from a log line. Each lines have about 1500 character.

I can only extract 2 fields and i get an error saying my rex has exceeded configured match_limit, consider raising the value in limits.conf.

Any suggestion of where I am doing wrong? Is that possible is my rex(as below) not right?

"^\d+\-\d+\-\d+\s+\d+:\d+:\d+\s+\[\w+\]\s\w+\s+\w+\s+\{\s+\[\w+\]\s+\w+\s+\w+\s+\{\s+(?P<ReadablePayload>.+)\s+\}\,\s+\[\w+\]\s+\w+\s+\{\s+(?P<Reason>.+\s+.+\s+.+\s+.+\s+.+\s+.+\s+.+\s+.+\s+.+\s+.+)\s+"

Here is an example of events :

2019-02-08 00:54:43 [TRACE] Malformed message { 
[TRACE] Readable Payload { UNH+20038142932000+AIRRQT:15:2:1A+cepSwzsAXFw=800'ORG+1A:MUC+1234567:NCE1A0950+++T'BLK+152+RF++000+0000000000+1A:1290861+001:001'AMD++07:00000000::07FEB++1A:1290861+1A:1290861+MUC:1A: +00:01+++AMSSG34AA:57211534:02+++++++++AMSSG34AA:57211534'AVC+IBERIA+IB:0755'BIN+TRFP'CSC+7906:/++9993WSSU'DAT+180130+190207+190207'HAS+000+000:X:OPO+XXX+MAD+XXX+XX:XXXXX:X:X: : : : +XXXX++0'HAS+000+000:X:MAD+XXX+PMI+XXX+XX:XXXXX:X:X: : : : +XXXX++0'HAS+000+000:X:PMI+XXX+MAD+XXX+XX:XXXXX:X:X: : : : +XXXX++0'HAS+000+000:X:MAD+XXX+OPO+XXX+XX:XXXXX:X:X: : : : +XXXX++0'RFD+L+30JAN18++EUR:88.00:88.00:0.00++XT:48.90:48.90'KFL'KRF+ +Q:EUR:3.47:PT: *Q:EUR:10.09:YP: *Q:EUR:25.92:JD: *Q:EUR:7.69:QV: *Q:EUR:1.73:OG: 'ING'INM+001+01:SANTOS/CARLA MS'TTN+E+075+5169861484'EQN+1'CPN+1*2*3*4'REF+075-5169861484+07FEB19'END+X'UNT+23+1' }, 
[TRACE] Reason { Mandatory element is missing. 
[TRACE] Error at <src/CBRSegmentDecoders.cpp:21405 
[TRACE] Error in segment CSC 06 1 1A TKA in element #2. 
[TRACE] ------ 
[TRACE] Error at <src/CBRMessageDecoders.cpp:121 
[TRACE] Error in group Group id: _294728_G_CBR in element #3. 
[TRACE] ------ 
[TRACE] Decoding error in message AIRRQT 15 2 1A PNR in element #4 at character 236 of the buffer, using charset B 
[TRACE] Converted segments: ORG - BLK - AMD - AVC - BIN - 
[TRACE] Cannot convert CSC from here==>+9993WSSU'Last segment correctly processed: CSC+7906:/++9993WSSU' 
[TRACE] }, 
[TRACE] Rejection type { T }, 
[TRACE] Current line { 251590 }, 
[TRACE] }
0 Karma
1 Solution

chrisyounger
SplunkTrust
SplunkTrust

Hi @clementros

You are receiving this error becuase your regular expression takes a lot of steps to try and match your events.

Give this one a try instead: "Readable Payload {\s*(?P<ReadablePayload>[^}]+)[^{]+{ (?P<Reason>[^}]+)"

When you are trying to develop a complex regular expression, pay attention to the "step count" in the top right of this screen:
https://regex101.com/r/vl1jHG/1

Its not only the step count of valid matches you need to worry about, you also need to worry about how many steps there are when it is not expected to match the content as well.

Finally, in case you are interested, I expect your regular expression is performing poorly becuase of this part: .+ This will first match up to the very end of the event. The regular expression engine will then realise there is more that it needs to try match (specifically \s+\}) and then it will start backtracking to try find where it can match those extra parts. This is very computationally expensive.

Hope this helps,
Chris.

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

What you want to extract from sample data you have provided ? If you write your regex effectively you will able to extract required field with least steps. For example I have used sample data which you provided and tried to extract data ( https://regex101.com/r/fznUMj/1 ) and it has only 64 steps, however your regex has more than 29K steps.

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi @clementros

You are receiving this error becuase your regular expression takes a lot of steps to try and match your events.

Give this one a try instead: "Readable Payload {\s*(?P<ReadablePayload>[^}]+)[^{]+{ (?P<Reason>[^}]+)"

When you are trying to develop a complex regular expression, pay attention to the "step count" in the top right of this screen:
https://regex101.com/r/vl1jHG/1

Its not only the step count of valid matches you need to worry about, you also need to worry about how many steps there are when it is not expected to match the content as well.

Finally, in case you are interested, I expect your regular expression is performing poorly becuase of this part: .+ This will first match up to the very end of the event. The regular expression engine will then realise there is more that it needs to try match (specifically \s+\}) and then it will start backtracking to try find where it can match those extra parts. This is very computationally expensive.

Hope this helps,
Chris.

clementros
Path Finder

Thank you ! It work well 🙂

0 Karma

splunk_abhishek
Explorer

@chrisyoungerjds Thanks mate! your query has helped me to resolve a similar issue as well.. Cheers

0 Karma