topic Re: How to detect not-reporting hosts? in Splunk Search

How to detect not-reporting hosts?

unitedmarsupial — Tue, 04 Feb 2020 22:27:36 GMT

We have a large number of hosts reporting to Splunk, and sometimes (rarely), some of them stop sending events. Is there an elegant search for hosts, which have last reported anything more than T ago?

I'd like to make an alert for T being above, say, 6 hours or so...

Re: How to detect not-reporting hosts?

mattymo — Tue, 04 Feb 2020 22:37:47 GMT

please check out "trackme" on Splunkbase by the amazing app by @guilmxm

https://splunkbase.splunk.com/app/4621/

Great app that helps you manage and alert on data sources!

Re: How to detect not-reporting hosts?

gjanders — Wed, 05 Feb 2020 06:57:11 GMT

Great app! If you want an alternative to that app try Broken Hosts or Meta Woot!

Re: How to detect not-reporting hosts?

gcusello — Wed, 05 Feb 2020 07:42:30 GMT

Hi @unitedmarsupials,
you have to create a lookup (e.g. called perimeter.csv with a field called host) containing all the hosts to monitor; then you have to run a search like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

In this way you have all the hosts from your list that didn't send logs in the monitoring period.
You can create an alert to run e.g. every 5 minutes.
If you delete the last row and add the row | eval status=if(total=0,"Missing","Up") you have a dashboard that display the host status.

Ciao.
Giuseppe

Re: How to detect not-reporting hosts?

unitedmarsupial — Wed, 05 Feb 2020 14:43:06 GMT

Thanks for the ideas, but why do I need to create a lookup? The hosts are already known to Splunk -- all those, that have reported in the last, say, 30 days, but have not reported in the last 5 hours.

Re: How to detect not-reporting hosts?

unitedmarsupial — Wed, 05 Feb 2020 14:47:54 GMT

Thanks, but I don't have the access necessary to install new apps...

Re: How to detect not-reporting hosts?

gcusello — Wed, 05 Feb 2020 15:05:45 GMT

Hi @unitedmarsupials,
A manually managed lookup is the easiest way to be sure about the monitoring perimeter: if you e.g. take the hosts of last 24 hours, you don't check hosts that didn't send in the last period!

Anyway, it this could be sufficient for you, you can schedule a search every night that populates the perimeter.csv lookup so you haven't to do nothing.

| metedata index=_internal earliest=-24h
| dedup host
| sort host
| table host
| outputlookup perimeter.csv

and then run the above search e.g. every 5 minutes.

Ciao.
Giuseppe

Re: How to detect not-reporting hosts?

unitedmarsupial — Wed, 05 Feb 2020 17:15:51 GMT

This is, what I ended up using -- thanks to @gcusello for the stats ... BY host idea:

a search for normal events
| fields host, _time
| stats max(_time) AS most_recent by host
| where most_recent < relative_time(now(), "-5h")
| eval most_recent = strftime(most_recent, "%F %T")

The above performs whatever search you typically use, then looks for hosts, that haven't reported any search-satisfying matches within the specified time (5 hours in the above example). The search time-range is set by the usual time-picker, which should, obviously, include the alert time.

(The relative_time call can, probably, be expressed nicer, but this works.)

Re: How to detect not-reporting hosts?

mattymo — Wed, 05 Feb 2020 20:51:56 GMT

yep! Honorable mention for meta woot for sure!

Re: How to detect not-reporting hosts?

mattymo — Wed, 05 Feb 2020 20:54:07 GMT

can't you just talk to the humans that do have access to install apps???

Much easier than you re-inventing the wheel. Also based on the question below about why a lookup is necessary, I would recommend you save the scars of learning 😉

Plus once your alert goes nuts...you'll see why the app is so cool

Re: How to detect not-reporting hosts?

gcusello — Thu, 06 Feb 2020 07:39:29 GMT

Hi @unitedmarsupials,
your solution surely solves your functional need, but I think that's a very slow search if you use _internal (this means that you cannot execute it in an alarm e.g. every five minutes!) and a not sure search if you use a different index (because it's possible that you don't have nothing to receive on that index!).
In addition, you don't check servers that didn't send logs in the search timeframe.

I used the above solutions for an alert (with a frequency of 5 minutes) that's running from many years!

Ciao and next time!
Giuseppe