topic Re: Alert or Show when a host is down from any index in Alerting

Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 16:45:12 GMT

I have multiple indexes that I would like to look across all of them see when a host stops sending logs or down within a 24 hour window.

Re: Alert or Show when a host is down from any index

somesoni2 — Mon, 28 Oct 2013 18:50:26 GMT

All the hosts (whether they are sending data or not) send heartbeat to indexer in _internal index. you can query that to identify if a host is down or not.

index=_internal source=*metrics.log group=tcpin_connections earliest=-7d@d
| eval sourceHost=coalesce(hostname, sourceHost)
| eval age = (now() - _time )
|stats first(age) as age, first(_time) as LastTime by sourceHost
| convert ctime(LastTime) as "Last Active On"
| eval Status= case(age < XXX,"Running",age > XXX,"DOWN")

Where XXX=duration in second for which is their are no heartbeat from host, the host is down. Typically is can be 2-3 min (120 or 180)

Re: Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 19:12:22 GMT

I tried this search out and wasn't able to yield any results. I tried changing the XXX to differents seconds and still didn't yield any results.

Re: Alert or Show when a host is down from any index

kristian_kolb — Mon, 28 Oct 2013 19:15:23 GMT

Or you can look at the metadata to see when a host last sent some data. The example below lists hosts that have not sent data in the last day (86400 seconds). Should be significantly quicker than searching through the metrics logs.

| metadata type=hosts |where recentTime < now() - 86400 | eval lastSeen = strftime(recentTime, "%F %T") | fields + host lastSeen

Re: Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 19:24:16 GMT

This worked fine, but it left out some host. There's a host that's not showing that I know isn't reporting. Hasn't been reporting for over 3 days now.

Re: Alert or Show when a host is down from any index

kristian_kolb — Mon, 28 Oct 2013 19:27:06 GMT

does it show with only the first part?

| metadata type=hosts

Re: Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 19:29:33 GMT

It shows with only the first part, but not with the rest.

Re: Alert or Show when a host is down from any index

kristian_kolb — Mon, 28 Oct 2013 19:31:15 GMT

Ok.

| metadata type=hosts | eval lastSeen = strftime(recentTime, "%F %T") | fields + host lastSeen

Re: Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 19:37:48 GMT

Ok, this one shows everything. It doesn't just show ones that are not loggin or down. Is it possible to show only the ones that are down or not logging.

Re: Alert or Show when a host is down from any index

kristian_kolb — Mon, 28 Oct 2013 20:14:03 GMT

well, yes, that is the purpose of the where statement in the original query. However, does lastSeen for your 'missing' host seem correct?

Re: Alert or Show when a host is down from any index

jaywilwk — Mon, 28 Oct 2013 20:18:13 GMT

Yes it worked.

Re: Alert or Show when a host is down from any index

thelen_m_kevin — Thu, 28 Aug 2014 15:05:12 GMT

I know this is a bit dated, but I was interested in finding hosts that "suddenly stop reporting to splunk" and I found this answer.

When I run this search everything looks fine, and it makes sense. But I decided to test this by issuing a stop command on one of my forwarding agents. That device no longer shows up in the list at all instead of showing up with a "down" status.

Can anyone take a stab at why that would happen? (I haven't altered the search except to add seconds where there are XXX's)

Re: Alert or Show when a host is down from any index

somesoni2 — Thu, 28 Aug 2014 15:30:34 GMT

Did it show the host before you put it down (with same search)? The query depends on existence of fields hostname OR sourceHost from the events from that host (its used in stats so if either of the field is null they won't show up.

Re: Alert or Show when a host is down from any index

splunker9999 — Thu, 10 Nov 2016 15:42:46 GMT

@somesoni2, do we to run your search on Indexer? When I tried running this on one of my search head, I was getting status of my splunk servers but not forwarders?