topic Re: Get the exception and Error in Splunk query in Splunk Search

Get the exception and Error in Splunk query

bhartiya008 — Tue, 23 Feb 2021 11:57:22 GMT

I am trying to build a splunk query to get the error summary from a log. I want to capture all the events where there is some ERROR, Exception or Failure.

Below is the sample data :

ERROR org.mule.component.ComponentException: Failed to invoke ScriptComponent{bapmFlow.component.797791858}. Component that caused exception is: ScriptComponent{bapmFlow.component.797791858}. host = host1 = /odt/mule_/logs/bapm.logsourcetype = gdt_index 2/7/21 12:00:04.000 AM 2021-02-07 00:00:04,422 [[Java2python].bapmFlow.stage1.03] ERROR org.mule.exception.CatchMessagingExceptionStrategy - Failed to dispatch message to error queue after it failed to process. This may cause message loss. Message identification summary here: id=54972f10-6901-11eb-ad2a-0050568f5886 correlationId=<not set>, correlationGroup=-1, correlationSeq=-1 host = host1 = /odt/mule_/logs/bapm.logsourcetype = gdt_index 2021-02-07 00:00:04,407 [[Java2python].bapmFlow.stage1.03] ERROR org.mule.exception.CatchMessagingExceptionStrategy - ******************************************************************************** Message : org.mule.module.db.internal.domain.connection.ConnectionCreationException: Cannot get connection for URL jdbc:sqlserver://VLTROUXRPT.us.global.crux.com\PRS:1713;databaseName=DFT;domain=US;integratedSecurity=false;authenticationScheme=JavaKerberos;userName=Jack;password=<<credentials>>;trustServerCertificate=true;encrypt=true; : Login failed for user 'Jack'. ClientConnectionId:34edad77-7de1-4d0f-bc13-0fb7f090f722 (java.sql.SQLException) 2021-02-07 00:00:02,936 [[Java2python].bapmFlow.stage1.03] ERROR org.mule.exception.CatchMessagingExceptionStrategy - ... 89 lines omitted ... 2021-02-07 00:00:02,951 [[Java2python].bapmFlow.stage1.03] ERROR org.mule.exception.CatchMessagingExceptionStrategy - Failed to dispatch message to error queue after it failed to process. This may cause message loss. Message identification summary here: id=54970800-6901-11eb-a3d3-0050568f5165 correlationId=<not set>, correlationGroup=-1, correlationSeq=-1

I have noticed the below: The ERROR keyword before the failures with the exception name. So I built this basic query like below but it's not giving the desired results:

index=hdt  sourcetype=gdt_index ("ERROR" AND "Exception") OR "FAILED"
| rex ".*?(?<Exception>(\w+\.)+\w*Exception).*"
| rex "(?<ErrorMessage>\"Message\":(.*\",))"
| stats values(ErrorMessage) as ErrorMessage by Exception

Re: Get the exception and Error in Splunk query

ITWhisperer — Tue, 23 Feb 2021 13:16:30 GMT

You don't have anything in your example that contains "Message": so ErrorMessage would not contain anything - what were you expecting it to hold?

Re: Get the exception and Error in Splunk query

bhartiya008 — Tue, 23 Feb 2021 14:31:54 GMT

Thanks @ITWhisperer Yes ..You are right. I was trying to follow the examples I had in my project.
I want the message of the failures which comes right after the exception
For e.g.

Failed to invoke ScriptComponent{bapmFlow.component.797791858}. Component that caused exception is: ScriptComponent{bapmFlow.component.797791858}.

Cannot get connection for URL jdbc:sqlserver://VLTROUXRPT.us.global.crux.com\PRS:1713;databaseName=DFT;domain=US;integratedSecurity=false;authenticationScheme=JavaKerberos;userName=Jack;password=<<credentials>>;trustServerCertificate=true;encrypt=true; : Login failed for user 'Jack'. ClientConnectionId:34edad77-7de1-4d0f-bc13-0fb7f090f722 (java.sql.SQLException)

I want the exception name and the messages with which it failed.

Re: Get the exception and Error in Splunk query

ITWhisperer — Tue, 23 Feb 2021 14:45:34 GMT

You can get both values from one rex expression - I extended the Exception part to include other words such as Strategy to get the complete name of the exception, then skip over the non-words (spaces, colons, etc.), then assume the remainder of the line was the error message you wanted.

| rex ".*?(?<Exception>(\w+\.)+\w*Exception\w*)\W+(?<ErrorMessage>.*)"

Re: Get the exception and Error in Splunk query

bhartiya008 — Tue, 23 Feb 2021 14:53:01 GMT

@ITWhisperer --This looks Perfect to me!!
Thanks !!

Re: Get the exception and Error in Splunk query

bhartiya008 — Tue, 23 Feb 2021 14:56:20 GMT

@ITWhisperer Can you also please explain a bit about it.

Re: Get the exception and Error in Splunk query

ITWhisperer — Tue, 23 Feb 2021 15:21:13 GMT

".*?(?<Exception>(\w+\.)+\w*Exception\w*)\W+(?<ErrorMessage>.*)" .*? - not really needed since * means 0 or more so could match anything or nothing (?<Exception>(\w+\.)+\w*Exception\w*) - first capture group <Exception> - name of field (\w+\.)+ - one or more groups of "letters" followed by a . e.g. class in exception class hierarchy \w*Exception - zero or more "letters" followed by Exception \w* - zero or more "letters" Strings which match this are put into the Exception field (assuming the rest of the expression matches) \W+ - one or more "non-letter" e.g. punctuation and spaces (?<ErrorMessage>.*) - second capture group <ErrorMessage> - name of field .* - zero or more of anything until end of the line

Re: Get the exception and Error in Splunk query

bhartiya008 — Tue, 23 Feb 2021 15:35:15 GMT

@ITWhisperer Thank you so much!! This will help 🙂