Optimized Rex

[ivoZgu]

Hey All,
Can anybody help me with optimization of this rex:
| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Example log:
"#HLS# IID: EB_FILE_S, STEP: SEND_TOF, PKEY: Ids:100063604006, 1000653604006, 6000125104001, 6000135104001, 6000145104001, 6000155104001, STATE: IN_PROGRESS, MSG0: Sending request to K, EXCID: dcd, PROPS: EVENT_TYPE: SEND_TO_S, asd: asd #HLE#

ERROR:
"Streamed search execute failed because: Error in 'rex' command: regex="#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#" has exceeded configured match_limit, consider raising the value in limits.conf."

[ITWhisperer]

Something is not quite right here

• Your regex string is missing some question marks (although they do appear to be in your error message!)
• Your error message says you have hit a limit with max_match, but your rex command doesn't appear to be using max_match and your sample log is a single line so even if you were using max_match there would only be one set of results

Please can you clarify / expand your question

[ivoZgu]

Hey @ITWhisperer ,
You are absolutely right, just have edited the rex .
I am not using max_match. This error appears once there are a lots of matching events. 
On the other hand rex with less steps and one property (exclude EXCID) less achieves to retrieve all events and no error thrown.

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Thanks!
BR,
Ivo

[ITWhisperer]

Try braking the large rex up into smaller chunks

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
and so on

[ivoZgu]

Thank you!
Unfortuantely

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*(?P<STATE>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*(?P<MSG0>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*[a-zA-Z_]+,\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

did not help much.

[ITWhisperer]

Try reducing the lines until the error goes away to find out where the breakpoint is