Splunk Search

How to crete an alert to notify when host back to normal?

smanojkumar
Contributor

I'm having a list of serve down and need to notify once its back to normal (up),  This is the requirement,

once the server is up, no need to consider the same server further, because its already up , need to check the remaining.

Eg.., There are servers A,B,C,D and E are down, which will be there in lookup,

Need to check those server every minute and notify once its up,

if server A,B is up after some time,  then it should trigger an alert, already server A,B is up, and after next alert, server A,B should not be considered, only remaining servers like C,D and E should be considered further, Then it check and trigger alert when C, D and E or either one is up.


index=linux sourcetype=df
| lookup Hobbit_threshold_data host mount outputnew l_threshold as lower_value h_threshold as higher_value condition as Condition
| where ((PercentUsedSpace >= lower_value) AND (PercentUsedSpace<higher_value))
| where Condition!="no"
| eval hostname=mvindex(split(host,"."),0) [ | inputlookup Hobbit_Disk_Space_Warning.csv | fields host ]
| stats host=lower(host)
| stats count BY host
| append [ | inputlookup KCI_Hobbit_Disk_Space_Warning.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| eval status=if(total=0,"Down","Up")

(Bolded query part gives you server down list host)
I just modified as per the query given by you,  It does not meet the requirement.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...