Alerting

Raise alert on status change only

atelesca
Explorer

Hello,
I would like to be able to raise an alert when the status of a condition changes?
For example:
- when the CPU user percentage is higher than 80% I would like to raise an alert that launch a command raising the alarm_on to an external program. This should be done only if the previous CPU user percentage was lower than 80%. There is no need to raise the alarm again.
- when the CPU user percentage goes back to below 80% I would like to send a command to my external program that set the alarm_off. This has to be done only if the previous CPU user percentage was higher than 80%.
Do you know how can I do it in Splunk?
I could raise an alert based on a search such as
sourcetype="cpu"| where CPU="all" and pctUser>20
and add the conditions to
sourcetype="cpu"| where CPU="all" and pctUser>20 | stats count=1
so that I am sure that the condition has been met only once.
The problem is:
- I should specify the time, like last 2 minutes
- How can I make sure that this is met by host?

Thanks for your help.
Cheers,
Adriana

Tags (1)
0 Karma

atelesca
Explorer

Well, at the end the only way I found is to create one alert that raise the alarm when the following condition is met:

sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c>80

that runs every minute and it looks at the past 2 minutes.

In order to unraise the alarm, I created another alert that is launched when the following condition is met:
sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c<80

it also runs every minute and looks at the past 2 minutes.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Index This | What travels the world but is also stuck in place?

April 2026 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Discover New Use Cases: Unlock Greater Value from Your Existing Splunk Data

Realizing the full potential of your Splunk investment requires more than just understanding current usage; it ...

Continue Your Journey: Join Session 2 of the Data Management and Federation Bootcamp ...

As data volumes continue to grow and environments become more distributed, managing and optimizing data ...