topic How can we detect excessive overlapping alerts? in Deployment Architecture

How can we detect excessive overlapping alerts?

ddrillic — Thu, 20 Dec 2018 17:02:13 GMT

We reach situations in which application teams set their alerts at the top of the hour and when we (the Splunk team) catch it, it might be too late.

Is there a way to produce a report which lists the run times and detect excessive usage times?

Re: How can we detect excessive overlapping alerts?

skoelpin — Thu, 20 Dec 2018 19:00:15 GMT

Yeah, you can use the internal index for this. You should explicitly add savedsearch_name for this

index=_internal savedsearch_name=*
| timechart max(run_time) AS run_time by savedsearch_name

Re: How can we detect excessive overlapping alerts?

ddrillic — Thu, 20 Dec 2018 21:41:19 GMT

Thank you @skoelpin.

I changed the max to sum and we can see -

We can see that at each quarter of the hour we have peak usage.
Can we find out from _internal how many searches were skipped?

Re: How can we detect excessive overlapping alerts?

skoelpin — Thu, 20 Dec 2018 21:46:43 GMT

Yes, you sure can!

index=_internal sourcetype=scheduled status=skipped NOT "_ACCELERATE*"
| timechart count by savedsearch_name

Re: How can we detect excessive overlapping alerts?

ddrillic — Thu, 20 Dec 2018 21:56:28 GMT

Just ran -

index=_internal sourcetype=scheduler status=skipped NOT "_ACCELERATE*"
 | timechart count

It shows -

Re: How can we detect excessive overlapping alerts?

ddrillic — Thu, 20 Dec 2018 21:57:35 GMT

The totals for an hour are -

Re: How can we detect excessive overlapping alerts?

skoelpin — Thu, 20 Dec 2018 22:09:02 GMT

Yeah, you have a problem with skips at 4am. You should trend this over time by using timewrap to see if there's a pattern. Most likely, other searches are competing for resources and they run long and cause skips. You can fix this by changing search priroty away from 0 to auto.

You can split by savedsearch_name or get a total over a span of time by adding span=1h. We use this search to alert us and cut a ticket when we start skipping. Skips are unacceptable for us

Re: How can we detect excessive overlapping alerts?

ddrillic — Fri, 21 Dec 2018 20:39:31 GMT

Much appreciated @skoelpin.