Alerting

Alert based on total for only some rows

jdibble
Explorer

I have a search set up to display HTTP status results with totals and percentage of the total events.

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | fields - total

I need to set up an alert for this search based on the results of only some of the columns using a custom condition. I would like to send an alert only if the percentage of the total results is greater than 1% for the combined 404 and 500 errors. I need to know what custom condition search would enable me to do this.

Tags (2)

jdibble
Explorer

Aside from the solution nick posted, I came up with an alternative. (it also includes all 400 and 500 errors, not just 404 and 500) Here's the base search:

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | search (sc_status=4* OR sc_status=5*) | eventstats sum(count) as errortotal | eval errper = round(100*(errortotal/total),2) | fields - errortotal, total

With an alert condition of

search errper > 1

This filters it down to just the errors, and the only problem I have with it is that it keeps a column I don't want to display (errper) because removing it from the fields causes the alert not to fire. Still, it does what I need it to do so I can work on finding a way to hide that one column later.

0 Karma

sideview
SplunkTrust
SplunkTrust

I'm sure it's possible with the stats command, and I can update this answer when I have more time, but if it's all the same to you, the custom condition search would be easier if the underlying search was using chart count over foo by sc_status instead of stats count by sc_status.

sourcetype="iis" | eval foo=1 | chart count over foo by sc_status | addtotals | eval percent = round(100*(count/Total),2) . " %" | fields - Total

this trick with chart will put all the data into a single row so it can be much more easily tinkered with later. For instance your custom condition would be just

where 100 * (404+500) > Total

UPDATE::::

If you need it to be a stats search, then I was able to get this working as a custom condition search. I did however need to drop the "%" character from your percent field, to get the percents to add up though. With some time and this as an example I'm sure you can figure out the best way though.

| search sc_status=404 OR sc_status=500 | fields percent | transpose | addtotals | where Total>1

0 Karma

sideview
SplunkTrust
SplunkTrust

OK. I updated my answer. It's possible but clunky.

0 Karma

jdibble
Explorer

Thanks for the answer, but I need to do it with the stats command if it's at all possible.

0 Karma
Get Updates on the Splunk Community!

New Case Study: How LSU’s Student-Powered SOCs and Splunk Are Shaping the Future of ...

Louisiana State University (LSU) is shaping the next generation of cybersecurity professionals through its ...

Splunk and Fraud

Join us on November 13 at 11 am PT / 2 pm ET!Join us for an insightful webinar where we delve into the ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...