Splunk Search

Problem with optimization of the regex: limits.conf

spisiakmi
Communicator

Hi I have a problem with the error message of the Splunk: Error in 'rex' command: regex="(?ms)]+^\s\" has exceeded configured match_limit, consider raising the value in limits.conf
The problem is, that the regex regex="(?ms)\<test[^\>]+[^\s](?P<tmp>.*?)\</test\>"for + xml file generates 8099 steps.
I tested it on this xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<unitData  endtime="2019-09-30T05:39:08+02:00"  equipment="eq1"  equipmentClass="eqc1"  locale="german"  operator="ADMINISTRATOR"  senderID="sender1"  starttime="2019-09-30T05:38:09+02:00"  state="nok"  unit="74375513159930675"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:noNamespaceSchemaLocation="unitData-1.1.xsd">
   <subUnitData  position="1"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930675">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="2"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930676">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="3"  positionType="Panel Nr."  state="ok"  subUnit="74375513159930678">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="passed">
         <subTest  name="Verbindung zum Pr³fling aufbauen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Initialisierung"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Fehlerflag lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ID lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Sachnummer lesen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="Trigger setzen"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
         <subTest  name="ADC Abschalten"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
   <subUnitData  position="4"  positionType="Panel Nr."  state="nok"  subUnit="74375513159930677">
      <test  description="A10-007 7437551"  name="CU102_BAY2_QR"  testResultCode="failed">
         <subTest  name="FLOAT"  testPosition="unknown">
            <subPositions>
               <subPosition  name="{27}"/>
            </subPositions>
            <subTestResult  testResultClass="fail"  testResultCode="failed"/>
         </subTest>
         <subTest  name="Components not tested"  testPosition="unknown">
            <subTestResult  testResultClass="pass"  testResultCode="passed"/>
         </subTest>
      </test>
   </subUnitData>
</unitData>

Can you help me, please, to optimize the regex? I want to extract the test tag.

0 Karma
1 Solution

ololdach
Builder

Hi, take a look at spath. It might be the better solution to extract the fields: https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Spath

View solution in original post

0 Karma

ololdach
Builder

Hi, take a look at spath. It might be the better solution to extract the fields: https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Spath

0 Karma

spisiakmi
Communicator

Hi ololdach,

Thank you for your message.
I used the spath, but I lost the data relation. E.g. In CSV file there is a relation data model guarantied by the first row (header). How would I select here all test name where testResultCode is failed? Using spath of course. And than using the spath the subTest names, which are failed, depending on testname?

0 Karma

spisiakmi
Communicator

Hi ololdach,

I used the spath and it worked. I had only problem with mvexpand, because of the error message: command.mvexpand: output will be truncated at 300 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.
I solved it, with the command: |fields - _*
But thank you anyway.

0 Karma

ololdach
Builder

hi spisiakmi, glad I could help!

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi spisiakmi,
try this regex

(?ms)\<test\s(?P<tmp>.*?)\<\/test\>

that you can test at https://regex101.com/r/HHTNrR/1

Bye.
Giuseppe

0 Karma

spisiakmi
Communicator

Hi Giuseppe,

thank you, but your regex generates more steps, than mine. Mine has 8099, but your 8871. Sorry.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...