topic How to detect a host which stops sending logs? in Getting Data In

How to detect a host which stops sending logs?

bbiandov — Tue, 21 Oct 2014 00:46:50 GMT

Hi everyone, How do I detect a host which stops sending logs? Ideally an alert would be the optimal application of an elaborate search. So far |metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime doesn't do the trick.

It only works for one day and then it stops producing results (we are assuming the host which stopped forwarding stays that way meaning it worked once upon a time and then it stopped working; it doesn't just start working again by itself)...hence the need to detect this condition MORE THAN a day after the host stopped forwarding. For example I want the search to produce result if a host last forwarded a log 28 days ago and since then it hasn't forwarded anything.

Thank you!

Re: How to detect a host which stops sending logs?

gkanapathy — Tue, 21 Oct 2014 04:10:00 GMT

Seems like you should just be able to increase the number you're comparing against age, e.g., to 86400 x 28. I suppose if you want to every day report on every host that has ever stopped sending, then you can just remove that filter entirely.

Re: How to detect a host which stops sending logs?

bbiandov — Tue, 21 Oct 2014 04:31:39 GMT

Thanks gkanapathy, the challenge I am facing is that the search works just fine; but if saved as an alert one can ONLY select 1 day; sure you can change it in the GUI but once saved it reverts back to 1 day. That's what breaks it:

https://www.dropbox.com/s/2dzon27ssowqicg/splunk.png?dl=0

Re: How to detect a host which stops sending logs?

gkanapathy — Tue, 21 Oct 2014 04:34:15 GMT

you need to edit the search, not the time selector.

Re: How to detect a host which stops sending logs?

linu1988 — Tue, 21 Oct 2014 06:47:10 GMT

you need to run over 2/3 days to get the missing forwarder alert again and again. The problem with daily alert is metdata doesn't show you older records.

Re: How to detect a host which stops sending logs?

jd — Wed, 08 Jul 2015 09:10:13 GMT

You can use an app named : "deployement monitor"