topic Re: Optimized Rex in Splunk Enterprise

Optimized Rex

ivoZgu — Fri, 13 Sep 2024 07:50:24 GMT

Hey All,
Can anybody help me with optimization of this rex:
| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Example log:
"#HLS# IID: EB_FILE_S, STEP: SEND_TOF, PKEY: Ids:100063604006, 1000653604006, 6000125104001, 6000135104001, 6000145104001, 6000155104001, STATE: IN_PROGRESS, MSG0: Sending request to K, EXCID: dcd, PROPS: EVENT_TYPE: SEND_TO_S, asd: asd #HLE#

ERROR:
"Streamed search execute failed because: Error in 'rex' command: regex="#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#" has exceeded configured match_limit, consider raising the value in limits.conf."

Re: Optimized Rex

ITWhisperer — Thu, 12 Sep 2024 13:08:57 GMT

Something is not quite right here

Your regex string is missing some question marks (although they do appear to be in your error message!)
Your error message says you have hit a limit with max_match, but your rex command doesn't appear to be using max_match and your sample log is a single line so even if you were using max_match there would only be one set of results

Please can you clarify / expand your question

Re: Optimized Rex

ivoZgu — Fri, 13 Sep 2024 14:06:46 GMT

Hey @ITWhisperer ,
You are absolutely right, just have edited the rex .
I am not using max_match. This error appears once there are a lots of matching events.
On the other hand rex with less steps and one property (exclude EXCID) less achieves to retrieve all events and no error thrown.

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Thanks!
BR,
Ivo

Re: Optimized Rex

ITWhisperer — Fri, 13 Sep 2024 09:02:09 GMT

Try braking the large rex up into smaller chunks

Re: Optimized Rex

ivoZgu — Fri, 13 Sep 2024 19:19:10 GMT

Thank you!
Unfortuantely

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*(?P<STATE>[^,]+),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*(?P<MSG0>.*?),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*.*#HLE#" | rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*[a-zA-Z_]+,\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

did not help much.

Re: Optimized Rex

ITWhisperer — Fri, 13 Sep 2024 15:42:34 GMT

Try reducing the lines until the error goes away to find out where the breakpoint is