Problem with optimization of the regex: limits.conf

spisiakmi — Sat, 05 Oct 2019 11:03:31 GMT

Hi I have a problem with the error message of the Splunk: Error in 'rex' command: regex="(?ms)]+^\s\" has exceeded configured match_limit, consider raising the value in limits.conf
The problem is, that the regex regex="(?ms)\<test[^\>]+[^\s](?P<tmp>.*?)\</test\>"for + xml file generates 8099 steps.
I tested it on this xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<unitData  endtime="2019-09-30T05:39:08+02:00"  equipment="eq1"  equipmentClass="eqc1"  locale="german"  operator="ADMINISTRATOR"  senderID="sender1"  starttime="2019-09-30T05:38:09+02:00"  state="nok"  unit="74375513159930675"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:noNamespaceSchemaLocation="unitData-1.1.xsd">
   <subUnitData  position="1"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930675">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="2"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930676">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="3"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930678">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="4"  positionType="Panel Nr."  state="nok"  subUnit="74375513159930677">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="failed">
         <subTest  name="FLOAT"  testPosition="unknown">
            <subPositions>
               <subPosition  name="{27}"/>
            </subPositions>
            <subTestResult  testResultClass="fail"  testResultCode="failed"/>
         </subTest>
         <subTest  name="Components not tested"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
</unitData>

Can you help me, please, to optimize the regex? I want to extract the test tag.

Re: Problem with optimization of the regex: limits.conf

gcusello — Sat, 05 Oct 2019 11:41:58 GMT

Hi spisiakmi,
try this regex

(?ms)\<test\s(?P<tmp>.*?)\<\/test\>

that you can test at https://regex101.com/r/HHTNrR/1

Bye.
Giuseppe

Re: Problem with optimization of the regex: limits.conf

spisiakmi — Sat, 05 Oct 2019 12:41:53 GMT

Hi Giuseppe,

thank you, but your regex generates more steps, than mine. Mine has 8099, but your 8871. Sorry.

Re: Problem with optimization of the regex: limits.conf

ololdach — Sat, 05 Oct 2019 17:13:59 GMT

Hi, take a look at spath. It might be the better solution to extract the fields: https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Spath

Re: Problem with optimization of the regex: limits.conf

spisiakmi — Mon, 07 Oct 2019 06:47:07 GMT

Hi ololdach,

Thank you for your message.
I used the spath, but I lost the data relation. E.g. In CSV file there is a relation data model guarantied by the first row (header). How would I select here all test name where testResultCode is failed? Using spath of course. And than using the spath the subTest names, which are failed, depending on testname?

Re: Problem with optimization of the regex: limits.conf

spisiakmi — Wed, 30 Sep 2020 02:31:22 GMT

Hi ololdach,

I used the spath and it worked. I had only problem with mvexpand, because of the error message: command.mvexpand: output will be truncated at 300 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.
I solved it, with the command: |fields - _*
But thank you anyway.

Re: Problem with optimization of the regex: limits.conf

ololdach — Fri, 11 Oct 2019 13:06:32 GMT

hi spisiakmi, glad I could help!

topic Re: Problem with optimization of the regex: limits.conf in Splunk Search

Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf

Re: Problem with optimization of the regex: limits.conf