Quickly find the time since the last event logged by a host

[approachct]

We are trying to monitor the hosts to ensure they have not stopped logging events. The search being used is

*|stats max(EVENT_TIMESTAMP) AS LastTime by WEB_SERVER
 |convert timeformat="%Y%m%d %H:%M:%S" mktime(LastTime)
 |eval MinutesAgo=now()-LastTime
 |convert timeformat="%M:%S" ctime(MinutesAgo)
 |fields WEB_SERVER, MinutesAgo
 |sort -MinutesAgo
 |table WEB_SERVER, MinutesAgo

I expect that there is a much more efficient way of doing this, probably going after some of the metadata that is stored in Splunk along with the log event.

Any ideas?

[piebob · Accepted solution]

this search looks at the recentTime and provides a table of hosts sorted by the last time they were heard from:

| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time

or, you could enable the Deployment Monitor app and configure it to alert you when forwarders haven't been heard from:

http://www.splunk.com/base/Documentation/latest/Deploy/Troubleshootyourdeployment