topic Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf". in Splunk Search

Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Wed, 28 Nov 2018 09:04:03 GMT

"ListOfLineItems.*?<OrderFunction>(?<OrderFunction>[^<]*)"

Want to know if any alternative regex can be written to avoid this error.

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

FrankVl — Wed, 28 Nov 2018 09:14:04 GMT

Can you share sample data to which this is applied? And some more description on what you want to accomplish?

The problem is likely with the .*? part, but without further info it is impossible to suggest better alternatives.

To extract fields from XML data, you may also want to consider using the spath command instead of rex: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Wed, 28 Nov 2018 11:20:51 GMT

Thanks Frank.

Below is the sample xml. Here, I want to extract OrderFunction which comes immediately after ListOfLineItems.(bcoz in the single xml, many OrderFunctions are present)

<SiebelMessage MessageId="#####" MessageType="Integration Object" IntObjectName="Order Interface" IntObjectFormat="Siebel Hierarchical" TransactionName="SubmitOrder"><ListOfOrderInterface><Orders operation=""><Id>#####</Id><RelationalOrderId>######</RelationalOrderId><BillToAccountId>#####</BillToAccountId><IntegrationId/><OrderNumber>#####</OrderNumber><OrderPriority>Medium</OrderPriority><OrderType>Change Proposition</OrderType><RequestedDate/><Status/><ARII>Australia</ARII><ReasonCode/><PaymentMethod>Postpay</PaymentMethod><ServiceProvider>Vodafone AU</ServiceProvider><OrganizationName>Vodafone AU</OrganizationName><BillingEffectDateTime>11/28/2018 11:10:02</BillingEffectDateTime><EffectDateTime>11/28/2018 11:10:02</EffectDateTime><OldBillingAccountId/><NewBillingAccountId/><CustomerAccountId>#####</CustomerAccountId><TargetServiceProvider/><OrderReasonCode>CP1 - Customer Requested</OrderReasonCode><tesaCode/><ConnectionType/><WBIReasonCode/><ToolsOfTrade>N</ToolsOfTrade><CustomerBillCycle>1</CustomerBillCycle><MigratedAccountFlag>N</MigratedAccountFlag><OrderFunction>UPGRADE</OrderFunction><AdditionalOrderFunction/><CustomerMainEmailAddr/><CustomerSegment>Consumer</CustomerSegment><CustType>Person</CustType><SiebelProductType>Voice</SiebelProductType><SpecialSiebelType/><BulkFlag/><ServiceOrderPriority>1</ServiceOrderPriority><ServicePositionIndicator/><CustomerType>Person</CustomerType><BAMarketingPreferencesFlag>Y</BAMarketingPreferencesFlag><SecondarySubs>N</SecondarySubs><SecondarySubsDisconnect>N</SecondarySubsDisconnect><SSReasonCode/><ListOfSDIMEI/><NBNAccessTechnology/><ByPassPassProvisioning>N</ByPassPassProvisioning><ListOfAdditionalInfo><AdditionalInfo><Type>Hard Caps</Type><Unthrottle/><UnthrottleSharing/><EndDate/><Product>IDD</Product><Status>Unbarred</Status><Action/></AdditionalInfo><AdditionalInfo><Type>Hard Caps</Type><Unthrottle/><UnthrottleSharing/><EndDate/><Product>Data</Product><Status>Unbarred</Status><Action/></AdditionalInfo></ListOfAdditionalInfo><ExternalReferenceId/><OrderChannel/><SalesPerson>NAVGIREAS</SalesPerson><ListOfLineItems><LineItems><WBIReasonCode/><EstDeliveryDt/><EstShipmentDt/><StockBand/><SpecialSiebelType/><ListOfSIM><SIM><ListOfSIMDetails><SIMDetails><Name>PUK</Name><Value>39092201</Value></SIMDetails><SIMDetails><Name>PUK2</Name><Value/></SIMDetails><SIMDetails><Name>Profile</Name><Value>GA431A</Value></SIMDetails></ListOfSIMDetails><AssetNumber>89610300003114915201</AssetNumber></SIM></ListOfSIM><ListOfNewSIM/><ListOfNewIMSI/><ListOfIMSI><IMSI><AssetNumber>505038309061393</AssetNumber></IMSI></ListOfIMSI><Id>#####</Id><ActionCode>Deleted</ActionCode><AssetId/><AssetIntegrationId/><OrderReasonCode/><RelatedAssetIntegrationId/><ReasonCode/><OrderFunction>UPGRADE</OrderFunction><RootStatus>Submitted</RootStatus><OrderType>Change Proposition</OrderType><BasePrice/><ShippingGroup/><IntegrationId>######</IntegrationId><OrderHeaderId>2-SI3QRJ1X</OrderHeaderId><ParentOrderItemId/><Product>Vodafone PostPay Sim Only Plans Aug 2017</Product><ProductId>2-CNDUGEN</ProductId><RootOrderItemId>#####</RootOrderItemId><Status>Submitted</Status><AssetInternalIntegrationId>######</AssetInternalIntegrationId><ServiceInstanceFlag>N</ServiceInstanceFlag><ServiceType>SUBSCRIPTION</ServiceType><SiebelProductType>Voice</SiebelProductType><ServicePIN>2207</ServicePIN><Resource>61411857667</Resource><NewResource/><Type>Proposition</Type><SiebelPrice/><OverrideAmount/><OverrideReasonCode/><CreatedBy>NAVGIREAS</CreatedBy><LastUpdBy>NAVGIREAS</LastUpdBy><InstalledAssetName>Ms SANDRA BENNET</InstalledAssetName><SiebelOnly>N</SiebelOnly><ChargeDescription/><VFPeriodEndDate/><VFPeriodStartDate/><VFPcrfFlag/><RelatedServiceOrderId/><WaiveCharges>N</WaiveCharges><ListOfAttributes><Attributes><Id>######</Id><ActionCode>Deleted</ActionCode><Name>Call Barring PIN</Name><ObjectId>2-SI3QRJ8P</ObjectId><Value>1234</Value><IntegrationId/></Attributes><Attributes><Id>2-SI3QRJ8T</Id><ListOfLineItems>

I have to use the same regex in datamodel in Splunk. I have to look if i can use spath.

Please help if i can try anything other than .*?

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

FrankVl — Wed, 28 Nov 2018 11:49:14 GMT

With just that sample data it works fine in Splunk 7.2.1, when I run below. So I'm wondering if the sample you shared is actually representative of the case that causes you to get that error.

| makeresults 
| eval xml="<SiebelMessage MessageId=\"#####\" MessageType=\"Integration Object\" IntObjectName=\"Order Interface\" IntObjectFormat=\"Siebel Hierarchical\" TransactionName=\"SubmitOrder\"><ListOfOrderInterface><Orders operation=\"\"><Id>#####</Id><RelationalOrderId>######</RelationalOrderId><BillToAccountId>#####</BillToAccountId><IntegrationId/><OrderNumber>#####</OrderNumber><OrderPriority>Medium</OrderPriority><OrderType>Change Proposition</OrderType><RequestedDate/><Status/><ARII>Australia</ARII><ReasonCode/><PaymentMethod>Postpay</PaymentMethod><ServiceProvider>Vodafone AU</ServiceProvider><OrganizationName>Vodafone AU</OrganizationName><BillingEffectDateTime>11/28/2018 11:10:02</BillingEffectDateTime><EffectDateTime>11/28/2018 11:10:02</EffectDateTime><OldBillingAccountId/><NewBillingAccountId/><CustomerAccountId>#####</CustomerAccountId><TargetServiceProvider/><OrderReasonCode>CP1 - Customer Requested</OrderReasonCode><tesaCode/><ConnectionType/><WBIReasonCode/><ToolsOfTrade>N</ToolsOfTrade><CustomerBillCycle>1</CustomerBillCycle><MigratedAccountFlag>N</MigratedAccountFlag><OrderFunction>UPGRADE</OrderFunction><AdditionalOrderFunction/><CustomerMainEmailAddr/><CustomerSegment>Consumer</CustomerSegment><CustType>Person</CustType><SiebelProductType>Voice</SiebelProductType><SpecialSiebelType/><BulkFlag/><ServiceOrderPriority>1</ServiceOrderPriority><ServicePositionIndicator/><CustomerType>Person</CustomerType><BAMarketingPreferencesFlag>Y</BAMarketingPreferencesFlag><SecondarySubs>N</SecondarySubs><SecondarySubsDisconnect>N</SecondarySubsDisconnect><SSReasonCode/><ListOfSDIMEI/><NBNAccessTechnology/><ByPassPassProvisioning>N</ByPassPassProvisioning><ListOfAdditionalInfo><AdditionalInfo><Type>Hard Caps</Type><Unthrottle/><UnthrottleSharing/><EndDate/><Product>IDD</Product><Status>Unbarred</Status><Action/></AdditionalInfo><AdditionalInfo><Type>Hard Caps</Type><Unthrottle/><UnthrottleSharing/><EndDate/><Product>Data</Product><Status>Unbarred</Status><Action/></AdditionalInfo></ListOfAdditionalInfo><ExternalReferenceId/><OrderChannel/><SalesPerson>NAVGIREAS</SalesPerson><ListOfLineItems><LineItems><WBIReasonCode/><EstDeliveryDt/><EstShipmentDt/><StockBand/><SpecialSiebelType/><ListOfSIM><SIM><ListOfSIMDetails><SIMDetails><Name>PUK</Name><Value>39092201</Value></SIMDetails><SIMDetails><Name>PUK2</Name><Value/></SIMDetails><SIMDetails><Name>Profile</Name><Value>GA431A</Value></SIMDetails></ListOfSIMDetails><AssetNumber>89610300003114915201</AssetNumber></SIM></ListOfSIM><ListOfNewSIM/><ListOfNewIMSI/><ListOfIMSI><IMSI><AssetNumber>505038309061393</AssetNumber></IMSI></ListOfIMSI><Id>#####</Id><ActionCode>Deleted</ActionCode><AssetId/><AssetIntegrationId/><OrderReasonCode/><RelatedAssetIntegrationId/><ReasonCode/><OrderFunction>UPGRADE</OrderFunction><RootStatus>Submitted</RootStatus><OrderType>Change Proposition</OrderType><BasePrice/><ShippingGroup/><IntegrationId>######</IntegrationId><OrderHeaderId>2-SI3QRJ1X</OrderHeaderId><ParentOrderItemId/><Product>Vodafone PostPay Sim Only Plans Aug 2017</Product><ProductId>2-CNDUGEN</ProductId><RootOrderItemId>#####</RootOrderItemId><Status>Submitted</Status><AssetInternalIntegrationId>######</AssetInternalIntegrationId><ServiceInstanceFlag>N</ServiceInstanceFlag><ServiceType>SUBSCRIPTION</ServiceType><SiebelProductType>Voice</SiebelProductType><ServicePIN>2207</ServicePIN><Resource>61411857667</Resource><NewResource/><Type>Proposition</Type><SiebelPrice/><OverrideAmount/><OverrideReasonCode/><CreatedBy>NAVGIREAS</CreatedBy><LastUpdBy>NAVGIREAS</LastUpdBy><InstalledAssetName>Ms SANDRA BENNET</InstalledAssetName><SiebelOnly>N</SiebelOnly><ChargeDescription/><VFPeriodEndDate/><VFPeriodStartDate/><VFPcrfFlag/><RelatedServiceOrderId/><WaiveCharges>N</WaiveCharges><ListOfAttributes><Attributes><Id>######</Id><ActionCode>Deleted</ActionCode><Name>Call Barring PIN</Name><ObjectId>2-SI3QRJ8P</ObjectId><Value>1234</Value><IntegrationId/></Attributes><Attributes><Id>2-SI3QRJ8T</Id><ListOfLineItems>"
| rex field=xml "ListOfLineItems.*?<OrderFunction>(?<OrderFunction>[^<]*)"

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

DavidHourani — Wed, 28 Nov 2018 12:10:16 GMT

Hey payal,

If you're hitting a limit due to excessive matching try changing this setting in limits.conf :

[rex]
match_limit = <integer>
* Limits the amount of resources that are spent by PCRE
  when running patterns that will not match.
* Use this to set an upper bound on how many times PCRE calls an internal
  function, match(). If set too low, PCRE might fail to correctly match a pattern.
* Default: 100000

If you're using the rex command you can add the parameter max_match so that your regex matches more than one field, setting it to 0 means unlimited. See here : http://docs.splunk.com/Documentation/Splunk/7.2.1/SearchReference/Rex
in your case that would be something like :

.... | rex field=xml max_match=0 "<OrderFunction>(?<OrderFunction>[^<]+)"

Let me know how that works for you.

Cheers,
David

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Wed, 28 Nov 2018 13:02:41 GMT

Yes Frank..

When searching for a longer duration I am getting the error. For a shorter span there are no errors. So.. yes for a sample you might not find any error.

So.. is it like no other option other than increasing the limit in limits.conf?

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Wed, 28 Nov 2018 13:04:42 GMT

Thanks David.

I tried with max_match but no luck. I wanted to know if there are any more option which i can try before i change limits.conf.

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

FrankVl — Wed, 28 Nov 2018 13:20:29 GMT

You mean the number of events you're processing affects the issue? Or are the actual live events much bigger than the sample you posted?

One thing you could try, is extract the whole ListOfLineItems block in a first regex and then apply a second regex to that to extract the OrderFunction value from inside that field. Not sure if it would help, but worth a shot.

Alternatively just see if xpath or xmlkv commands can help you here.

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

DavidHourani — Wed, 28 Nov 2018 14:00:13 GMT

you could go for spath command, or indexed extractions using KV_MODE=xml but that's about it I guess.

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Sun, 02 Dec 2018 11:36:38 GMT

Thanks Frank,

I am trying to extract ListOfLineItems block.. but as per my regex it is capturing more than one ListOfLineItems block.

ListOfLineItems>(?<Bulk>[\sA-Za-z0-9/<>-]*</ListOfLineItems>)

Can you please help me in extracting each blocks seperately

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

payal23 — Sun, 02 Dec 2018 11:44:51 GMT

There are more than one ListOfLineItems in a single xml.

Re: Regex is showing error as "exceeded configured match_limit, consider raising the value in limits.conf".

FrankVl — Mon, 03 Dec 2018 08:53:36 GMT

Then maybe that is where your problem is and why it is complaining about too many matches. Any luck with one of the xml parsing commands (xpath / xmlkv)?

Regarding your regex for extracting the line items, perhaps try with a negative lookbehind:

ListOfLineItems\>(?<Bulk>.+(?<!\<\/ListOfLineItems\>))\<\/ListOfLineItems\>

That approach may also work for your original attempt:

ListOfLineItems.+(?<!\<\/ListOfLineItems\>)<OrderFunction>(?<OrderFunction>[^<]*)