Alert on anomalous growth of thruput per sourcetype

Path Finder

I want to create a scheduled search that will be able to trigger an alert if a there's a sourcetype that has a thruput much higher than the average for that sourcetype for a given time period.

Rational: be able eliminate license violations by identifying "peaking" sourcetypes in a timely manner

It's quite easy to see the peaks on a timechart, e.g.

index=_internal group=per_sourcetype_thruput | timechart per_second(kb) by series

Splunk Employee
Splunk Employee

Depending exactly what you're trying to do, there are many ways to do this. The simplest is to use trendline/streamstats:

index=_internal group=per_sourcetype_thruput 
| bucket _time span=5m
| stats sum(kb) as kb by _time,series
| makecontinuous _time span=5m 
| streamstats global=f current=f window=24
    avg(kb) as moving_avg_kb
    stdev(kb) as stdev_kb
  by series

And then alert conditionally:

| where kb > (moving_avg_kb+(0.5*stdev_kb))

Of course, there is much tuning on what you consider to be "allowable" peaks and over what period of time. (e.g., the above will not catch a source growing slowly over a two-hour or longer period) More complex requirements are going to require either more searches or much more complex ones.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...