Splunk Enterprise

Optimized Rex

ivoZgu
Explorer

Hey All,
Can anybody help me with optimization of this rex:
| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Example log:
"#HLS# IID: EB_FILE_S, STEP: SEND_TOF, PKEY: Ids:100063604006, 1000653604006, 6000125104001, 6000135104001, 6000145104001, 6000155104001, STATE: IN_PROGRESS, MSG0: Sending request to K, EXCID: dcd, PROPS: EVENT_TYPE: SEND_TO_S, asd: asd #HLE#

ERROR:
"Streamed search execute failed because: Error in 'rex' command: regex="#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#" has exceeded configured match_limit, consider raising the value in limits.conf."

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Something is not quite right here

  • Your regex string is missing some question marks (although they do appear to be in your error message!)
  • Your error message says you have hit a limit with max_match, but your rex command doesn't appear to be using max_match and your sample log is a single line so even if you were using max_match there would only be one set of results

Please can you clarify / expand your question

0 Karma

ivoZgu
Explorer

Hey @ITWhisperer ,
You are absolutely right, just have edited the rex .
I am not using max_match. This error appears once there are a lots of matching events. 
On the other hand rex with less steps and one property (exclude EXCID) less achieves to retrieve all events and no error thrown.

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"



Thanks!
BR,
Ivo

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try braking the large rex up into smaller chunks

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
and so on
0 Karma

ivoZgu
Explorer

Thank you!
Unfortuantely

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*(?P<STATE>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*(?P<MSG0>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*[a-zA-Z_]+,\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

did not help much.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try reducing the lines until the error goes away to find out where the breakpoint is

0 Karma
Get Updates on the Splunk Community!

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...