Splunk Search

How to crete an alert to notify when host back to normal?

smanojkumar
Contributor

I'm having a list of serve down and need to notify once its back to normal (up),  This is the requirement,

once the server is up, no need to consider the same server further, because its already up , need to check the remaining.

Eg.., There are servers A,B,C,D and E are down, which will be there in lookup,

Need to check those server every minute and notify once its up,

if server A,B is up after some time,  then it should trigger an alert, already server A,B is up, and after next alert, server A,B should not be considered, only remaining servers like C,D and E should be considered further, Then it check and trigger alert when C, D and E or either one is up.


index=linux sourcetype=df
| lookup Hobbit_threshold_data host mount outputnew l_threshold as lower_value h_threshold as higher_value condition as Condition
| where ((PercentUsedSpace >= lower_value) AND (PercentUsedSpace<higher_value))
| where Condition!="no"
| eval hostname=mvindex(split(host,"."),0) [ | inputlookup Hobbit_Disk_Space_Warning.csv | fields host ]
| stats host=lower(host)
| stats count BY host
| append [ | inputlookup KCI_Hobbit_Disk_Space_Warning.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")

(Bolded query part gives you server down list host)
I just modified as per the query given by you,  It does not meet the requirement.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

A Season of Skills: New Splunk Courses to Light Up Your Learning Journey

There’s something special about this time of year—maybe it’s the glow of the holidays, maybe it’s the ...

Announcing the Migration of the Splunk Add-on for Microsoft Azure Inputs to ...

Announcing the Migration of the Splunk Add-on for Microsoft Azure Inputs to Officially Supported Splunk ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI! Discover how Splunk’s agentic AI ...