How can I identify hosts that don't have any events over a 4-hour period and create an alert?

[glenngermiathen]

I want to identify any host that doesn't have any events over a four hour period and create an alert. Having trouble extracting the individual host.

index=ind1
| timechart span=4h count by host
| where count = 0
| table host count time

[gcusello]

Hi blacknight659,
you have to create a lookup containing all the hosts in your perimeter to monitor (e.g. perimeter.csv, with one column called "host") and then run a search like this

index=ind1 earliest=-4h latest=now
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host)  | fields host ]
| stats sum(count) AS Total BY host
| where Total=0
| table host

Deleting the row "|where Total=0" you can have a situation of your perimeter to display in a dashboard (also in graphic mode).

Bye.
Giuseppe

[lfedak_splunk]

Hey @glenngermiathen, if they solved your problem, remember to "√Accept" an answer to award karma points 🙂

[justinatpnnl · Accepted solution]

You could use the metadata command for type=hosts. Splunk keeps track of all hosts that have sent data, including the first and last time of the events it has received. I have a similar one I use for this. The query below shows the hosts that have sent data within the last 24 hours, but not within the last 4 hours.

| metadata type=hosts index=ind1
| where recentTime < relative_time(now(), "-4h") AND recentTime > relative_time(now(), "-24h") 

[glenngermiathen]

Exactly what I needed, thanks!

[blacknight659]

This is tough, because it is easier to look for something that is there rather than something that is not. Also, showing that over a timechart might not be easy.

I have a solution I would like for you to consider. I am not 100% sure it will work, but it would be worth testing. If your hosts don't change, then you could use a inputlookup and use a subsearch to find only the list of host you care about.

index=ind1 [|inputlookup hosts.csv | fields host]
| transaction host maxspan=4h
| rename linecount as LogCount
| stats count as count sum(LogCount)
| table host linecount
| fillnull value="null"

If this works, then you can make a search at the end of this to find all the "null" hosts.

I hope this helps.

[glenngermiathen]

Thanks for the suggestion! I thought about using the static lookup, but the challenge that creates is that it must be maintained. If new hosts are added that I am not aware of they will not be monitored.