Alerting

Need an Alert to monitor when the duration of downtime reaches one hour

bluemarvel
Path Finder

host Status=Offline OR Status=Online | search target="" | selfjoin Status | sort _time,target | table _time,target,Status,src,host | dedup 1 Status,target | rename target as Agent_Host | rename Status as Current_Status | rename src as Source_IP

the machines go offline then come back on, need to monitor for the duration of the downtime as well as maybe alert when the duration exceeds an hour

thanks in advance

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi
when you monitor a downtime you have to monitor all servers: up servers and down servers.
To be sure that all the monitored servers are checked you have to create a lookup (e.g. perimeter.csv) with all the servers to monitor and try something like this

index=your_index Status=Offline OR Status=Online
| transaction host startswith="Offline" endswith="Online"
| eval host=upper(host), count=1
| append [ search index=_internal NOT [ search  index=your_index Status=Offline OR Status=Online  | eval host=upper(host), count=10 | dedup host | fields host count ]
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host count ]
| stats sum(count) AS Total values(duration) AS duration BY host
| eval Status=case(count=0,"Server Down",count=1,"Downtime="+tostring(duration,"duration",count>2,"Server Up")
| table host Status

Bye.
Giuseppe

0 Karma

renjith_nair
Legend

You can use transaction command if there are multiple events

your search | transaction Host, "other common unique fields for these two transaction" startswith="Offline" endswith="Online"|timechart duration

or use stats

your search|stats first(_time) as End,last(_time) as Start by Status|eval Difference=End-Start|chart Difference
Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...