Alerting

Alert when there is a X% increase in all events during a given time period?

Explorer

I want to create an alert that will notify me when there is a X% increase in the total amount of events in a time period (ie X% frequency increase). I can't seem to find out how to connect this data to pull this all together, specifically I cannot figure out how to connect the frequency of events during current time period against the total average # events per day. Any help is greatly appreciated.

Influencer

Awesome. SO I can just change the
"earliest=" condition to be 30d to
check the count against the daily
average over the last month?

Eh, you have to do a tiny bit more for that. The search you gave would be comparing the count for a full 30 days to today's count. If you want an average of the daily count for the last 30 days you need to do something like:

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | bucket _time span=1d | stats count as count by _time | stats avg(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth

Influencer

Also, I'm assuming you are doing the last 30 days to smooth out any spikes that may have occurred. In which case, you may even want to use a median instead of an average.

i.e.

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | bucket _time span=1d | stats count as count by _time | stats median(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth
0 Karma

SplunkTrust
SplunkTrust

Here is one sample search for alert. For example, I am checking if the data logged for an index/sourcetype today has increated over 50% from the data indexed for same index/sourcetype yesterday.

index=indexA sourcetype=sourcetypeA earliest=-1d@d latest=@d | stats count as Yesterday | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*Yesterday

Setup an alert if number of events from above search > 0

Updated

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | timechart span=1d count | stats avg(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth

Explorer

Awesome. SO I can just change the "earliest=" condition to be 30d to check the count against the daily average over the last month?

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | stats count as Month | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*Month

0 Karma

Explorer

The whole point is so that we can be notified that A. There was a massive increase in # logged events. B. React to a sudden increase accordingly or at least look into it

0 Karma