All Apps and Add-ons

Regex data parsing using Delimiter comma "has exceeded the configured depth_limit, consider raising the value in limits.conf." ?

Hi,

while data parsing i'm using the delimiter section to parse my data at that time i get the error

alt text

when i try to extract the same log using the "Regular" option i get the following error

alt text

please advice

Regards,

0 Karma
1 Solution

Ultra Champion

If it is indeed just comma separated data, then why use regular expressions? And just out of curiosity, how did you end up with those regular expressions that were causing issues, did you let Splunk generate those somehow?

Just configure DELIMS based extractions along these lines:

props.conf

[yoursourcetype]
REPORT-comma-separated-fields = comma-separated-fields

transforms.conf

[comma-separated-fields]
DELIMS = ","
FIELDS = AUD_SEQ_NO,PACKAGE_NAME,SERVICE_NAME,AUDIT_TIME,EVENT_NAME,SHORT_TEXT,LONG_TEXT,AUDIT_DATA,CONSUMER_ID,MESSAGE_ID,CONTEXT_ID,USER_NAME,USER_ID,USER_CONTEXT,COMPANY_ID,VERSION,SESSION_ID,CHANNEL_ID,BUSINESSUNIT_ID,SERVER_NAME

View solution in original post

Ultra Champion

If it is indeed just comma separated data, then why use regular expressions? And just out of curiosity, how did you end up with those regular expressions that were causing issues, did you let Splunk generate those somehow?

Just configure DELIMS based extractions along these lines:

props.conf

[yoursourcetype]
REPORT-comma-separated-fields = comma-separated-fields

transforms.conf

[comma-separated-fields]
DELIMS = ","
FIELDS = AUD_SEQ_NO,PACKAGE_NAME,SERVICE_NAME,AUDIT_TIME,EVENT_NAME,SHORT_TEXT,LONG_TEXT,AUDIT_DATA,CONSUMER_ID,MESSAGE_ID,CONTEXT_ID,USER_NAME,USER_ID,USER_CONTEXT,COMPANY_ID,VERSION,SESSION_ID,CHANNEL_ID,BUSINESSUNIT_ID,SERVER_NAME

View solution in original post

Hi Frank,

Added to the above query, i would like to know about the limit.conf file you have shared me before, the file you have shared and the file i'm having in the splunk has the same values. So can you please advice me that which value i should change to increase the limit of regex in the system.

Regards,
Vigneshprasanna R

0 Karma

Ultra Champion

I don't think you want to change those values, unless you understand exactly why you are hitting those limits and increasing the limits is the only solution. Hitting those limits typically means there is something wrong with your configuration, which you should fix, rather than stretching the limits.

0 Karma

Hi Frank,

Once Again Thanks for the effort, Ya i added the data to the splunk and went to a log and used the option of "Extract Fields" with there i got two options to extract the fields they are "Delimiter " & "Regular" i tried both but ended up with the error i have posted above.

I'm new to splunk, so i have a some more doubts in .conf files, can you please guide me where i should create this .conf files you have added in the previous answer ??

And one more doubt is that i have many types of log in my system how to specify that this conf is for a specific index type ??

Thanks & Regards,
Vigneshprasanna R

0 Karma

Ultra Champion

I tested the GUI field extraction method using the Delimiter method and that works fine on the sample you shared. Can you perhaps share one or more screenshots showing how you performed those steps that led to the error?

The props.conf and transforms.conf should in this case go onto your search head(s). You could put them under etc/system/local (or add the config to the files already there). Better solution would be to put them in your own custom app directory under etc/apps/.

0 Karma

Ultra Champion

Looks like you're running into the limits of below 2 settings (from limits.conf spec):

[rex]
match_limit = <integer>
* Limits the amount of resources that are spent by PCRE
  when running patterns that will not match.
* Use this to set an upper bound on how many times PCRE calls an internal
  function, match(). If set too low, PCRE might fail to correctly match a pattern.
* Default: 100000

depth_limit = <integer>
* Limits the amount of resources that are spent by PCRE
  when running patterns that will not match.
* Use this to limit the depth of nested backtracking in an internal PCRE
  function, match(). If set too low, PCRE might fail to correctly match a pattern.
* Default: 1000

For your second case, I'm guessing the multiple [^,\n]* bits are causing some issues. Perhaps you can share some sample data and explain what you want to achieve, such that we can help tune your regex?

0 Karma

Hi Frank,

Thanks in advance 🙂

Sample Message :
1162340588,xx_xxx,xx.yy.eded.aasa.mka,2018-05-30 19:49:54.477,End,Service TokenService Operation getTokenList completed successfully.,NULL,NULL,user1,Id1njk23nj13jma,NULL,NULL,NULL,UNKNOWN,BS0020001,v1,NULL,NULL,NULL,ppoansdo12-st34.metest.local

Heading need to be provided:
AUDSEQNO PACKAGENAME SERVICENAME AUDITTIME EVENTNAME SHORTTEXT LONGTEXT AUDITDATA CONSUMERID MESSAGEID CONTEXTID USERNAME USERID USERCONTEXT COMPANYID VERSION SESSIONID CHANNELID BUSINESSUNITID SERVERNAME

0 Karma