Alerting

Alert based on total for only some rows

jdibble
Explorer

I have a search set up to display HTTP status results with totals and percentage of the total events.

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | fields - total

I need to set up an alert for this search based on the results of only some of the columns using a custom condition. I would like to send an alert only if the percentage of the total results is greater than 1% for the combined 404 and 500 errors. I need to know what custom condition search would enable me to do this.

Tags (2)

jdibble
Explorer

Aside from the solution nick posted, I came up with an alternative. (it also includes all 400 and 500 errors, not just 404 and 500) Here's the base search:

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | search (sc_status=4* OR sc_status=5*) | eventstats sum(count) as errortotal | eval errper = round(100*(errortotal/total),2) | fields - errortotal, total

With an alert condition of

search errper > 1

This filters it down to just the errors, and the only problem I have with it is that it keeps a column I don't want to display (errper) because removing it from the fields causes the alert not to fire. Still, it does what I need it to do so I can work on finding a way to hide that one column later.

0 Karma

sideview
SplunkTrust
SplunkTrust

I'm sure it's possible with the stats command, and I can update this answer when I have more time, but if it's all the same to you, the custom condition search would be easier if the underlying search was using chart count over foo by sc_status instead of stats count by sc_status.

sourcetype="iis" | eval foo=1 | chart count over foo by sc_status | addtotals | eval percent = round(100*(count/Total),2) . " %" | fields - Total

this trick with chart will put all the data into a single row so it can be much more easily tinkered with later. For instance your custom condition would be just

where 100 * (404+500) > Total

UPDATE::::

If you need it to be a stats search, then I was able to get this working as a custom condition search. I did however need to drop the "%" character from your percent field, to get the percents to add up though. With some time and this as an example I'm sure you can figure out the best way though.

| search sc_status=404 OR sc_status=500 | fields percent | transpose | addtotals | where Total>1

0 Karma

sideview
SplunkTrust
SplunkTrust

OK. I updated my answer. It's possible but clunky.

0 Karma

jdibble
Explorer

Thanks for the answer, but I need to do it with the stats command if it's at all possible.

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...