Splunk Enterprise

Optimized Rex

ivoZgu
Explorer

Hey All,
Can anybody help me with optimization of this rex:
| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

Example log:
"#HLS# IID: EB_FILE_S, STEP: SEND_TOF, PKEY: Ids:100063604006, 1000653604006, 6000125104001, 6000135104001, 6000145104001, 6000155104001, STATE: IN_PROGRESS, MSG0: Sending request to K, EXCID: dcd, PROPS: EVENT_TYPE: SEND_TO_S, asd: asd #HLE#

ERROR:
"Streamed search execute failed because: Error in 'rex' command: regex="#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#" has exceeded configured match_limit, consider raising the value in limits.conf."

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Something is not quite right here

  • Your regex string is missing some question marks (although they do appear to be in your error message!)
  • Your error message says you have hit a limit with max_match, but your rex command doesn't appear to be using max_match and your sample log is a single line so even if you were using max_match there would only be one set of results

Please can you clarify / expand your question

0 Karma

ivoZgu
Explorer

Hey @ITWhisperer ,
You are absolutely right, just have edited the rex .
I am not using max_match. This error appears once there are a lots of matching events. 
On the other hand rex with less steps and one property (exclude EXCID) less achieves to retrieve all events and no error thrown.

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*STEP:\s*(?P<STEP>[^,]+),\s*PKEY:\s*(?P<PKEY>.*?),\s*STATE:\s*(?P<STATE>[^,]+),\s*MSG0:\s*(?P<MSG0>.*?),\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"



Thanks!
BR,
Ivo

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try braking the large rex up into smaller chunks

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
and so on
0 Karma

ivoZgu
Explorer

Thank you!
Unfortuantely

| rex "#HLS#\s*IID:\s*(?P<IID>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*(?P<STEP>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*(?P<PKEY>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*(?P<STATE>[^,]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*(?P<MSG0>.*?),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*(?P<EXCID>[a-zA-Z_]+),\s*.*#HLE#"
| rex "#HLS#\s*IID:\s*[^,]+,\s*STEP:\s*[^,]+,\s*PKEY:\s*.*?,\s*STATE:\s*[^,]+,\s*MSG0:\s*.*?,\s*EXCID:\s*[a-zA-Z_]+,\s*PROPS:\s*(?P<PROPS>[^#]+)\s*#HLE#"

did not help much.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try reducing the lines until the error goes away to find out where the breakpoint is

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Kubernetes Alerts That Work Out of the Box with AutoDetect detectors in Splunk ...

A deployment rolls out slowly, one pod sits not ready for a few minutes, and whoever is on call gets an alert ...

Observability at Your Fingertips Directly From Splunk Platform—for Free

At .conf, we announced our roadmap to bring observability foundations to Splunk Platform. Bringing metrics and ...