Other Usage

How to split alerts by error type?

YatMan
Explorer

Sample events:

 

{ 
    errorClass:  NullPointerException  
    userId: 53
},
{ 
    errorClass:  IllegalArgumentException  
    userId: 65
},
{ 
    errorClass:  NullPointerException  
    userId: 32
}

 

Instead of having one alert mixed with many different errors, I would like to create a new alert for every type of error:
Alert 1

 

{ 
    errorClass:  NullPointerException  
    userId: 53
},
{ 
    errorClass:  NullPointerException  
    userId: 32
}

 

Alert 2

 

{ 
    errorClass:  IllegalArgumentException  
    userId: 65
}

 


Of course the number of errorClass can be infinite so we cannot create a dedicated alert for every errorClass. I want to try if something like:

{baseSearch} |stats by errorClass
and every row in the stats table can be a new alert is possible.

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @YatMan,

you should try the option in alert creation "Trigger for each result" and run a search like this:

{baseSearch} 
| stats count values(userId) AS userId BY errorClass
| where count>3

the last condition is mandatory if you need a threshold, otherwise you can avoid to use it.

if you want the error count for each userId, you can put this field in the BY clause:

{baseSearch} 
| stats count BY errorClass userId 
| where count>3

Ciao.

Giuseppe

YatMan
Explorer

Thank you,
using | stat and"Trigger for each result" did send me unique alert for each error!

Is there a way I can attach the events in json format associated with each error in this alert?
I don't need to see the graph, I want to see the list of associated events along with the alert. 
The sample events I provided are of way simplified, we need the complete log (including stack trace, request Id , etc..) for onCall. I feel we are so close.

Screenshot 2023-06-22 at 21.19.39.png

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @YatMan,

you could add to the stats command, using the values option, the fields you need.

To see the row events is just a little complicated: you should use table comman instead stats and put the threshold in the alert editing.

Ciao.

Giuseppe

0 Karma

YatMan
Explorer

Thanks for the reply. The reason we went with stats is so that we can group by error type. Not sure if the same can be done with table commands.

With the table approach, essentially what we are looking for is :

Screenshot 2023-06-26 at 10.54.03.png

We create a table with 2 rows, so we get 2 alerts.
But each row has sub rows with the fields extracted from the events. The reason is so we can track which user at what time and what API etc

This also looks complicated, could you kindly point out if I am on the right direction?

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...