Alerting

Alert when there is a X% increase in all events during a given time period?

alvarezmj
Explorer

I want to create an alert that will notify me when there is a X% increase in the total amount of events in a time period (ie X% frequency increase). I can't seem to find out how to connect this data to pull this all together, specifically I cannot figure out how to connect the frequency of events during current time period against the total average # events per day. Any help is greatly appreciated.

masonmorales
Influencer

Awesome. SO I can just change the
"earliest=" condition to be 30d to
check the count against the daily
average over the last month?

Eh, you have to do a tiny bit more for that. The search you gave would be comparing the count for a full 30 days to today's count. If you want an average of the daily count for the last 30 days you need to do something like:

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | bucket _time span=1d | stats count as count by _time | stats avg(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth

masonmorales
Influencer

Also, I'm assuming you are doing the last 30 days to smooth out any spikes that may have occurred. In which case, you may even want to use a median instead of an average.

i.e.

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | bucket _time span=1d | stats count as count by _time | stats median(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth
0 Karma

somesoni2
Revered Legend

Here is one sample search for alert. For example, I am checking if the data logged for an index/sourcetype today has increated over 50% from the data indexed for same index/sourcetype yesterday.

index=indexA sourcetype=sourcetypeA earliest=-1d@d latest=@d | stats count as Yesterday | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*Yesterday

Setup an alert if number of events from above search > 0

Updated

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | timechart span=1d count | stats avg(count) as DailyAvgOfMonth | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*DailyAvgOfMonth

Stephan
Engager

Hello,
how can I do this for multiple hosts seperatly?
In my case I want to look for increasing Windows Events for each server.
regards
Stephan

0 Karma

Stephan
Engager

thanks. got it.

used innerjoin instead of "appendcols" and I have to set "timechart ... limit=0"

0 Karma

StringBee
Explorer

Can you please mention query here, it will be helpful...

Tags (1)
0 Karma

alvarezmj
Explorer

Awesome. SO I can just change the "earliest=" condition to be 30d to check the count against the daily average over the last month?

index=indexA sourcetype=sourcetypeA earliest=-30d@d latest=@d | stats count as Month | appendcols [search index=indexA sourcetype=sourcetypeA earliest=@d latest=now | stats count as Today ] | where Today>=1.5*Month

0 Karma

alvarezmj
Explorer

The whole point is so that we can be notified that A. There was a massive increase in # logged events. B. React to a sudden increase accordingly or at least look into it

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...