topic Re: Reduce the regex steps in Splunk Search

Reduce the regex steps

dpatiladobe — Mon, 15 Jun 2020 19:50:48 GMT

Trying to extract Dimensions out of Query, but it is taking 1500 plus steps due to which I am getting limits.conf error.

[{(, ](?<Dimensions>[a-z0-9A-Z\[\.\]+[\-\ \_]*)[\.&\[] SELECT { [Measures].[IMS Org Count] } ON COLUMNS, NONEMPTY ( { [End User].[End User ID].[End User ID].MEMBERS * [End User].[End User Name].[End User Name].MEMBERS * [Product].[PMBU Short Desc].[PMBU Short Desc].MEMBERS * [Product].[PMBU Medium Desc].[PMBU Medium Desc].MEMBERS * [IMS Org].[Unique Id].[Unique Id].MEMBERS * [IMS Org].[IMS Org Id].[IMS Org Id].MEMBERS * [IMS Org].[MC Org Name].[MC Org Name].MEMBERS * [Is Active Account].[Is Active Account].[Is Active Account].MEMBERS* [Billing End User].[End User ID].[End User ID].MEMBERS } , [Measures].[IMS Org Count] ) ON ROWS FROM ( SELECT CASE '1' WHEN "5" THEN [Account Manager].[AM Org Lead Ldap].[AM Org Lead Ldap].[xxxx] WHEN "4" THEN [Account Manager].[Regional Manager Ldap].[Regional Manager Ldap].[xxx] WHEN "3" THEN [Account Manager].[AM Manager Ldap].[AM Manager Ldap].[xxx] WHEN "2" THEN [Account Manager].[AM Lead Ldap].[AM Lead Ldap].[xxx] WHEN "1" THEN [Account Manager].[AM Ldap].[AM Ldap].[xxxx] END ON 0 FROM XX )

https://regex101.com/r/HEdUhy/1/

Re: Reduce the regex steps

cmerriman — Mon, 15 Jun 2020 15:52:07 GMT

Is the regex101 what you want extracted? all those groups? or is it not exactly what you want? what is the limit problem? is this a regex you're doing in props or transforms or are you doing it with |rex on the search line? can you provide the entire stanza or the entire search string?

Re: Reduce the regex steps

dpatiladobe — Mon, 15 Jun 2020 16:37:30 GMT

The issue is regex is taking almost ~1300 steps due to which i am getting limit error's.

The query is as below and regex is `[{(, ](?<Dimensions>[a-z0-9A-Z\[\.\]+[\-\ \_]*)[\.&\[]`

SELECT { [Measures].[IMS Org Count] } ON COLUMNS, NONEMPTY ( { [End User].[End User ID].[End User ID].MEMBERS * [End User].[End User Name].[End User Name].MEMBERS * [Product].[PMBU Short Desc].[PMBU Short Desc].MEMBERS * [Product].[PMBU Medium Desc].[PMBU Medium Desc].MEMBERS * [IMS Org].[Unique Id].[Unique Id].MEMBERS * [IMS Org].[IMS Org Id].[IMS Org Id].MEMBERS * [IMS Org].[MC Org Name].[MC Org Name].MEMBERS * [Is Active Account].[Is Active Account].[Is Active Account].MEMBERS* [Billing End User].[End User ID].[End User ID].MEMBERS } , [Measures].[IMS Org Count] ) ON ROWS FROM ( SELECT CASE '1' WHEN "5" THEN [Account Manager].[AM Org Lead Ldap].[AM Org Lead Ldap].[xxxx] WHEN "4" THEN [Account Manager].[Regional Manager Ldap].[Regional Manager Ldap].[xxx] WHEN "3" THEN [Account Manager].[AM Manager Ldap].[AM Manager Ldap].[xxx] WHEN "2" THEN [Account Manager].[AM Lead Ldap].[AM Lead Ldap].[xxx] WHEN "1" THEN [Account Manager].[AM Ldap].[AM Ldap].[xxxx] END ON 0 FROM XX )

Re: Reduce the regex steps

cpetterborg — Mon, 15 Jun 2020 18:04:28 GMT

I think that @cmerriman has a valid point about your data. Is that the exact set of matches that you want to get? If it is, then that makes it so that we know the actual constraints. If you are saying only that you want to get that data exactly as it is with that data, but you just want to reduce the steps so that you don't have to make changes to limits.conf, then that is one thing.

Looking at the results you got in regex101, I'm not seeing any kind of correlation to useful data that you could use from the regex you are using. I think that is your biggest problem in getting a good answer. If you want to get different results, then that would make finding a better regex a whole lot easier. As it is, I'm finding this very difficult to help find you a solution because if the data changes slightly but still be valid SQL, then you won't get a similar resulting set of Dimensions from your regex. Hopefully you can see the point I'm trying to make.

Re: Reduce the regex steps

dpatiladobe — Mon, 15 Jun 2020 18:07:28 GMT

Yes these what i wanted "If you are saying only that you want to get that data exactly as it is with that data, but you just want to reduce the steps so that you don't have to make changes to limits.conf, then that is one thing."

Re: Reduce the regex steps

cpetterborg — Mon, 15 Jun 2020 18:37:00 GMT

This regex will reduce your steps from 1322 to 1228, but I have no idea if it will prevent your limits errors:

[{(, ](?<Dimensions>[-.\w[\]+ ]+)[.&[]

The data that you are working with and the result set that you want is rather abstract and hard to make better, since it isn't very comprehendible. The actual goal of what a Dimensions is is not clear.

Re: Reduce the regex steps

Birbal — Tue, 16 Jun 2020 14:59:40 GMT

[{(, ](?<Dimensions>(\[[^\]]+\]\.)+)

gives 755 steps for the data provided in your regex101.com my solution at https://regex101.com/r/HEdUhy/2 with your exact prefix.

If you remove prefix it turns to be 357 steps for your given data.

(?<Dimensions>(\[[^\]]+\]\.)+)

solution at https://regex101.com/r/HEdUhy/3

Re: Reduce the regex steps

dpatiladobe — Tue, 16 Jun 2020 20:30:10 GMT

These is what something i wanted. Thank you Appreciated.