topic Re: locating dead hosts in Splunk Search

locating dead hosts

rbellini — Tue, 20 Nov 2012 17:26:43 GMT

I am trying to get a report of all hosts that have not reported any events in the last 2 hours. I am using:

| metadata type=hosts | search totalCount >0 AND lastTime < now-2h

It only reports hosts that have a totalCount >0 yet it ignores the lastTime qualifier.

Any suggestions?

Thank you!

Re: locating dead hosts

yannK — Tue, 20 Nov 2012 18:30:30 GMT

My method is to save a list of the hosts with a scheduled lookup, that updated every week
then use it in a join to compare to the list of the hosts who sent data in the last hour.

generate the lookup with last week hosts
earliest=-7d@d latest=-14d@d, scheduled to run every week

* | stats count AS previouscount by host | outputlookup host_previousweek.csv

scheduled test to compare to current hosts. (like last 2 hours)

| inputlookup host_previousweek.csv | JOIN type=outer host [ search * | stats count AS currentcount by host ] | eval check_test=if(isnull(currentcount),"missing",if(isnull(previouscount),"new","present")) | WHERE check_test="missing"
And alert on missing.
If you need to update the host lookup, run it manually

Re: locating dead hosts

jonuwz — Tue, 20 Nov 2012 18:34:20 GMT

| metadata type=hosts totalCount >0 | where lastTime < relative_time(now(),"-2h")

now() is only available in eval and where clauses.

You could not use relative_time() and just instead use now()-(60*60*2)

Re: locating dead hosts

rbellini — Wed, 21 Nov 2012 13:49:59 GMT

This almost worked! I made one minor change:

| metadata type=hosts | where lastTime < relative_time(now(),"-2h") AND totalCount >0

Thank you!

Re: locating dead hosts

rbellini — Wed, 21 Nov 2012 13:57:22 GMT

This also worked:

| metadata type=hosts | where lastTime <(now()-7200) AND totalCount >0

Re: locating dead hosts

rbellini — Wed, 21 Nov 2012 19:48:07 GMT

This presents a totall new concept on locating missing hosts!

Thanks!