topic Re: Alert when host stops reporting data (IT Essentials Learn) in Splunk Search

Alert when host stops reporting data (IT Essentials Learn)

jackjack — Thu, 02 Dec 2021 00:33:03 GMT

I am attempting to use a search from IT Essentials Learn named "Alert when host stops reporting data - Linux - IT Essentials Work"

Is it possible to filter this alert by host type? I've performed a number of tests now and it seems my only option is to search against all hosts.

Here is the search from IT Essentials Learn

I modified the 2 index lines and the host line. If I use * for all 3 it kind of works but checks against every host.

If I use host=*dev* it displays all hosts without the name *dev* as evaluating to 0 whereas all the *dev* hosts get evaluated to 1.

To counteract this I tried adding a where host=*dev* elsewhere (I tried it in the metadata portion, as a where clause at the end of all the metadata piping, as a where clause next to where val=0, etc.) but this has the effect of just completely removing a host that isn't sending data from the list (or removing all hosts), so that also does not work.

Is it possible to split this up based on hosts or am I stuck with an all or nothing?

Edit: I tried adding a where all the way at the end.

It does not work with host="*dev*"

However, I can use host!="some host name" to filter those out.

I'm not sure why I can use negation but not wildcards?

Edit2: I am searching on the prior 5mins if that matters at all

Re: Alert when host stops reporting data (IT Essentials Learn)

isoutamo — Thu, 02 Dec 2021 14:47:48 GMT

this issue is already hanlded couple of times 😉

Here is one old post where is links to other articles / ways to handle this quite efficient.

https://community.splunk.com/t5/Getting-Data-In/How-do-I-check-to-see-if-a-couple-of-Hosts-are-sending-data-to/m-p/544060

r. Ismo

Re: Alert when host stops reporting data (IT Essentials Learn)

jackjack — Thu, 02 Dec 2021 17:28:37 GMT

I always perform a search before asking a question. I did not find my specific question answered.

I am able to search across ALL hosts however my issue specifically is that I cannot figure out how to split them by host name.

I looked through the post you linked and this is the only link that may be helpful: https://www.duanewaddle.com/proving-a-negative/

However that is still a fundamentally different process using lookup tables and it's performing a count instead of looking at the last time sent.

Are you suggesting that my method, which is presented by splunk itself through the IT Essentials Learn App, will not work? Can I not filter using something similar to one of the methods I attempted in my original post?

Re: Alert when host stops reporting data (IT Essentials Learn)

jackjack — Fri, 10 Dec 2021 20:19:56 GMT

I found a solution so I'll post it. Unfortunately it doesn't solve the filtering issue. But it's good enough for me.

I ended up splitting out my index into dev and prod groups so I was able to solve this by simply specifying the index (the index performs all the filtering I need).

This search was helpful for me to determine which index to use in the command

|tstats dc(host) as val max(_time) as _time where index=* host=* by host, index

Then I simply modified the two index lines to use the index that I saw referenced in the above search.