Alerting

How to set up an alert for unresponsive hosts with different time allowances for different sourcetypes?

glenrattay
Engager

I have an alert that will trigger if a host does not respond for 60 minuets. I would like to be able to be able to specify sourcetypes so that we can have different allowances. e.g 1 day, 6 hours and 60 minuets.

| metadata type=hosts | eval responsive=if(recentTime < now() - 60*60*3, "no", "yes") | eval defunct=if(recentTime > now() - 60*60*24*3, "no", "yes") | where responsive="no" | convert ctime(recentTime) as last_contact | fields host, responsive, defunct, last_contact | rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

Any advice on how I can achieve this?

0 Karma

lguinn2
Legend

Well, your first problem is that the metadata command does not return any information about sourcetypes when you specify the type of hosts. I guess you could do it like this, assuming that 60 minutes is the minimum time for all sourcetypes:

index=* [ | metadata type=hosts | where recentTime < (now() - 3600) | fields host ]
| stats latest(_time) as lastTime by host sourcetype
| eval responsive = case(sourcetype="A" AND lastTime < (now() - 3600),"no",
                  sourcetype="B" AND lastTime < (now() - 86400),"no",
                  etc.,
                 1==1,"yes")
| where responsive="no" 
| eval defunct=if(lastTime > now() - 60*60*24*3, "no", "yes") 
| eval last_contact = strftime(lastTime,"%m/%d/%Y %X")
| sort host sourcetype
| table host, sourcetype, responsive, defunct, last_contact 
| rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

This is going to be a lot slower than your original search. You could set up a lookup table that contains the sourcetypes and the time allowance for each sourcetype. That will eliminate the complex case statement, but it won't make the search any faster.

0 Karma
Get Updates on the Splunk Community!

Index This | Why did the turkey cross the road?

November 2025 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  &#x1f680; Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Feel the Splunk Love: Real Stories from Real Customers

Hello Splunk Community,    What’s the best part of hearing how our customers use Splunk? Easy: the positive ...