Alerting

How to set up an alert for unresponsive hosts with different time allowances for different sourcetypes?

glenrattay
Engager

I have an alert that will trigger if a host does not respond for 60 minuets. I would like to be able to be able to specify sourcetypes so that we can have different allowances. e.g 1 day, 6 hours and 60 minuets.

| metadata type=hosts | eval responsive=if(recentTime < now() - 60*60*3, "no", "yes") | eval defunct=if(recentTime > now() - 60*60*24*3, "no", "yes") | where responsive="no" | convert ctime(recentTime) as last_contact | fields host, responsive, defunct, last_contact | rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

Any advice on how I can achieve this?

0 Karma

lguinn2
Legend

Well, your first problem is that the metadata command does not return any information about sourcetypes when you specify the type of hosts. I guess you could do it like this, assuming that 60 minutes is the minimum time for all sourcetypes:

index=* [ | metadata type=hosts | where recentTime < (now() - 3600) | fields host ]
| stats latest(_time) as lastTime by host sourcetype
| eval responsive = case(sourcetype="A" AND lastTime < (now() - 3600),"no",
                  sourcetype="B" AND lastTime < (now() - 86400),"no",
                  etc.,
                 1==1,"yes")
| where responsive="no" 
| eval defunct=if(lastTime > now() - 60*60*24*3, "no", "yes") 
| eval last_contact = strftime(lastTime,"%m/%d/%Y %X")
| sort host sourcetype
| table host, sourcetype, responsive, defunct, last_contact 
| rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

This is going to be a lot slower than your original search. You could set up a lookup table that contains the sourcetypes and the time allowance for each sourcetype. That will eliminate the complex case statement, but it won't make the search any faster.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...