topic Re: Grouping the messages by Exception message present in stack_trace field of a json event in Splunk Search

How to group the messages by exception message present in stack_trace field of a json event?

ghostrider — Tue, 11 Oct 2022 13:59:46 GMT

I have json events/messages in my search result. There is a field or property called "stack_trace" in the json like below. I want to group the events and count them as shown below based on the Exception Reason or message. The problem is traces are multi lined and hence below query that I am using is, it seems not able to extract the exact exception message. Is there a way to achieve the expected output?

Event

{ MESSAGE : Failed to send stack_trace : com.abc.xyz.package.ExceptionName: Missing A. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79) }

Query I am using

MY_SEARCH | rex field=stack_trace "(?<exceptionclass>\w+): (?<exceptiontext>\w+)." | stats count as Count by "exceptiontext"

Expected Output

Exception Count Missing A 3 Missing B 4 Missing C 1

Re: Grouping the messages by Exception message present in stack_trace field of a json event

gcusello — Tue, 11 Oct 2022 11:49:26 GMT

Hi @ghostrider,

you have to review your regex, please try this:

MY_SEARCH | rex field=stack_trace "ExceptionName: (?<exceptiontext>[^\.]+)" | stats count as Count by "exceptiontext"

that you can test at https://regex101.com/r/OAJ4Iw/1

Ciao.

Giuseppe

Re: Grouping the messages by Exception message present in stack_trace field of a json event

ghostrider — Tue, 11 Oct 2022 13:10:56 GMT

Thank you!!. Issue is "ExceptionName" this is not same for all the exceptions. Is there a way to completely ignore this field and just get the exception message?

Re: Grouping the messages by Exception message present in stack_trace field of a json event

gcusello — Tue, 11 Oct 2022 13:29:31 GMT

Hi @ghostrider,

sorry but I don't understand you need: maybe you should categorize the messages to find all the regexes to extract the message.

If you could share a sample of the other kind of logs I could try to find a regex.

Ciao.

Giuseppe

Re: Grouping the messages by Exception message present in stack_trace field of a json event

ghostrider — Tue, 11 Oct 2022 14:13:33 GMT

Ok. So I am trying to say is currently we can have different exceptionnames in the events like below. In this case your query will not work since you are matching the ExceptionName literally. So is there any way to ignore the entire text till ":" and just extract the "Missing A" etc part?

Event 1

{ MESSAGE : Failed to send stack_trace : com.abc.xyz.package.ExceptionName: Missing A. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79) }

Event 2

{ MESSAGE : Failed to send stack_trace : com.abc.xyz.package.OtherExceptionName: Missing B. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79) }

{ MESSAGE : Failed to send stack_trace : com.abc.xyz.package.SomeOtherExceptionName: Missing C. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79) }

Event 3

Re: Grouping the messages by Exception message present in stack_trace field of a json event

gcusello — Tue, 11 Oct 2022 14:27:04 GMT

Hi @ghostrider,

you have to identify the string to capture.

Viewing your sample with the same regex, you can take all the messages as you can see at https://regex101.com/r/OAJ4Iw/2

It's not relevant if before ExceptionName there something else, it's important that there's "ExceptionName: ".

If you haven't this word it's difficoult because you have many colons in your logs so it isn't sufficient to identify the string to capture.

Ciao.

Giuseppe

Re: Grouping the messages by Exception message present in stack_trace field of a json event

ghostrider — Wed, 12 Oct 2022 06:53:55 GMT

Yes thanks your query is perfect. Just was curious is there any way to include in the regex one condition to extract the string till the current line only and not go to next line? Currently you are having "." as the limiting char till which we can read the string

Re: Grouping the messages by Exception message present in stack_trace field of a json event

gcusello — Wed, 12 Oct 2022 09:30:31 GMT

Hi @ghostrider,

sorry but I don't understand, could you share and highlit what you want to extract?

Ciao.

Giuseppe

Re: Grouping the messages by Exception message present in stack_trace field of a json event

PickleRick — Wed, 12 Oct 2022 09:57:03 GMT

You have to anchor your regex somewhere 🙂

Otherwise the regex processor will not know where to start or stop. That's why the constant part of ExceptionName. Regex is a simple tool which matches strings to patterns, it doesn't understand "business logic" and cannot guess what you want 😉

So you have to either anchor it with a specific constant term(s) or restrict it to a special pattern. There's no way around it. You could try extracting, for example, a second line from each matching event, but then you'd have to be sure it's always on the second line.

Re: Grouping the messages by Exception message present in stack_trace field of a json event

yuanliu — Thu, 13 Oct 2022 05:45:46 GMT

@ghostrider have you tried just using

| rex field=stack_trace "^(?<classname>.+)\.\w+: (?<exceptiontext>.+)"

Using your sample data, output is like

classname	exceptiontext	stack_trace
com.abc.xyz.package	Missing A.	com.abc.xyz.package.ExceptionName: Missing A. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79)

By default, rex stops at the first line.