Alerting

Raise alert on status change only

atelesca
Explorer

Hello,
I would like to be able to raise an alert when the status of a condition changes?
For example:
- when the CPU user percentage is higher than 80% I would like to raise an alert that launch a command raising the alarm_on to an external program. This should be done only if the previous CPU user percentage was lower than 80%. There is no need to raise the alarm again.
- when the CPU user percentage goes back to below 80% I would like to send a command to my external program that set the alarm_off. This has to be done only if the previous CPU user percentage was higher than 80%.
Do you know how can I do it in Splunk?
I could raise an alert based on a search such as
sourcetype="cpu"| where CPU="all" and pctUser>20
and add the conditions to
sourcetype="cpu"| where CPU="all" and pctUser>20 | stats count=1
so that I am sure that the condition has been met only once.
The problem is:
- I should specify the time, like last 2 minutes
- How can I make sure that this is met by host?

Thanks for your help.
Cheers,
Adriana

Tags (1)
0 Karma

atelesca
Explorer

Well, at the end the only way I found is to create one alert that raise the alarm when the following condition is met:

sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c>80

that runs every minute and it looks at the past 2 minutes.

In order to unraise the alarm, I created another alert that is launched when the following condition is met:
sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c<80

it also runs every minute and looks at the past 2 minutes.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...