topic Re: Capture groups extracting empty values from log messages in Splunk Search

Capture groups extracting empty values from log messages

search_in_splun — Wed, 03 Apr 2024 14:47:55 GMT

Requesting help with search query. I have application logs in Splunk like,

2024-04-02T12:26:02.244-04:00,severity=DEBUG,thread=main,logger=org.apache.catalina.core.NamingContextListener,{},Creating JNDI naming context
2024-04-02T12:26:02.118-04:00,severity=DEBUG,thread=main,logger=org.apache.catalina.core.NamingContextListener,{}, Adding resource ref UserDatabase ResourceRef[className=org.apache.catalina.UserDatabase,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=description,content=User database that can be updated and saved},{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.apache.catalina.users.MemoryUserDatabaseFactory},{type=pathname,content=conf/tomcat-users.xml}]

And I'm using following query to separate different sections of the message,

index=my_app_index AND source="**/my-app-service.log" AND sourcetype="app_v1"|rex="(?<mydatetime>^\S*)\,severity=(?<severity>\S*)\,thread=(?<thread>\S*)\,logger=(?<logger>\S*)\,\{\}\,(?<logmsg>(.)*)"|table mydatetime,logger,thread,_raw,logmsg|rename logmsg AS MESSAGE

What I see is,

column mydatetime and logmsg(MESSAGE) are empty.

What I expect is,

column mydatetime contain initial date-time, and logmsg(MESSAGE) contain the last message part

mydatetime	logger	thread	logmsg
2024-04-02T12:26:02.244-04:00	org.apache.catalina.core.NamingContextListener	main	Creating JNDI naming context
2024-04-02T12:26:02.118-04:00	org.apache.catalina.core.NamingContextListener	main	Adding resource ref UserDatabase ResourceRef[className=org.apache.catalina.UserDatabase,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=description,content=User database that can be updated and saved},{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.apache.catalina.users.MemoryUserDatabaseFactory},{type=pathname,content=conf/tomcat-users.xml}]

Re: Capture groups extracting empty values from log messages

ITWhisperer — Wed, 03 Apr 2024 21:02:57 GMT

You don't need the = after the rex

| rex "(?<mydatetime>^\S*)\,severity=(?<severity>\S*)\,thread=(?<thread>\S*)\,logger=(?<logger>\S*)\,\{\}\,(?<logmsg>.*)"

Updated to remove brackets in the logmsg pattern

Re: Capture groups extracting empty values from log messages

search_in_splun — Wed, 03 Apr 2024 15:41:41 GMT

Yes indeed it does solve the issue, but now there's a new issue

Streamed search execute failed because: Error in 'rex' command: regex="(?<mydatetime>^\S*)\,severity=(?<severity>\S*)\,thread=(?<thread>\S*)\,logger=(?<logger>\S*)\,\{\}\,(?<logmsg>(.)*)" has exceeded the configured depth_limit, consider raising the value in limits.conf..

Re: Capture groups extracting empty values from log messages

ITWhisperer — Wed, 03 Apr 2024 16:17:22 GMT

Again, what's with the = after the regex? Is this just a typo?

Re: Capture groups extracting empty values from log messages

richgalloway — Wed, 03 Apr 2024 16:17:59 GMT

This regex works with the sample events and is much more efficient according to regex101.com.

| rex "(?<mydatetime>[^,]+),severity=(?<severity>[^,]+),thread=(?<thread>[^,]+),logger=(?<logger>[^,]+),\{\},(?<logmsg>.*)"

Re: Capture groups extracting empty values from log messages

search_in_splun — Wed, 03 Apr 2024 16:23:39 GMT

I re-checked by putting the rex you've provided once again without the equal(=) symbol, but surprisingly the error message comes back with words 'regex='

Re: Capture groups extracting empty values from log messages

search_in_splun — Wed, 03 Apr 2024 16:26:07 GMT

And this rex doesn't produce any error