topic Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert? in Splunk Search

How can I identify hosts that don't have any events over a 4-hour period and create an alert?

glenngermiathen — Mon, 09 Oct 2017 21:10:39 GMT

I want to identify any host that doesn't have any events over a four hour period and create an alert. Having trouble extracting the individual host.

index=ind1
| timechart span=4h count by host
| where count = 0
| table host count time

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

blacknight659 — Mon, 09 Oct 2017 21:48:56 GMT

This is tough, because it is easier to look for something that is there rather than something that is not. Also, showing that over a timechart might not be easy.

I have a solution I would like for you to consider. I am not 100% sure it will work, but it would be worth testing. If your hosts don't change, then you could use a inputlookup and use a subsearch to find only the list of host you care about.

If this works, then you can make a search at the end of this to find all the "null" hosts.

I hope this helps.

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

justinatpnnl — Mon, 09 Oct 2017 21:56:42 GMT

You could use the metadata command for type=hosts. Splunk keeps track of all hosts that have sent data, including the first and last time of the events it has received. I have a similar one I use for this. The query below shows the hosts that have sent data within the last 24 hours, but not within the last 4 hours.

| metadata type=hosts index=ind1
| where recentTime < relative_time(now(), "-4h") AND recentTime > relative_time(now(), "-24h")

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

lfedak_splunk — Tue, 10 Oct 2017 00:27:07 GMT

Hey @glenngermiathen, if they solved your problem, remember to "√Accept" an answer to award karma points 🙂

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

gcusello — Tue, 10 Oct 2017 07:41:01 GMT

Hi blacknight659,
you have to create a lookup containing all the hosts in your perimeter to monitor (e.g. perimeter.csv, with one column called "host") and then run a search like this

index=ind1 earliest=-4h latest=now
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=upper(host)  | fields host ]
| stats sum(count) AS Total BY host
| where Total=0
| table host

Deleting the row "|where Total=0" you can have a situation of your perimeter to display in a dashboard (also in graphic mode).

Bye.
Giuseppe

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

glenngermiathen — Tue, 10 Oct 2017 13:58:46 GMT

Thanks for the suggestion! I thought about using the static lookup, but the challenge that creates is that it must be maintained. If new hosts are added that I am not aware of they will not be monitored.

Re: How can I identify hosts that don't have any events over a 4-hour period and create an alert?

glenngermiathen — Tue, 10 Oct 2017 15:55:18 GMT

Exactly what I needed, thanks!