topic Re: How do I make an alert for hosts that haven't sent their logs in? in Alerting

How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Thu, 28 May 2020 23:31:31 GMT

I am only curious for a certain index

index=abc | stats count by host | stats sum(count) AS Total BY host | where Total>0

This search is good to see how many logs are coming in for my hosts in that index but the problem is when a host stops sending I have no alert for it. I tried changing the "|where Total>=0" but it took off the host from my table when it hit zero. How can I adjust or change my query to make it so I can alert when a host hits 0 logs.

Re: How do I make an alert for hosts that haven't sent their logs in?

to4kawa — Fri, 29 May 2020 04:02:57 GMT

index=abc 
| stats count by host 
| append [| makeresults
| eval host=split("hostA,hostB,hostC ... ",",")
| mvexpand host
| fields host
| table host]
| stats sum(count) AS Total BY host | where Total>0

instead of append [....] , you can create host.csv and use | inputlookup append=t host.csv

Re: How do I make an alert for hosts that haven't sent their logs in?

gjanders — Fri, 29 May 2020 10:04:02 GMT

If you instead are looking for a more global solution consider:
TrackMe
Or Meta Woot!
Or Broken hosts

I use TrackMe at the moment 🙂

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Fri, 29 May 2020 21:11:32 GMT

using this lookup table would it just have one column that being my hosts and the servers underneath it with no other columns?

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Fri, 29 May 2020 22:18:57 GMT

Thank you, my concern is does this app need to go on my forwarders or the is the app downloaded to app folder on my splunk indexer?

Re: How do I make an alert for hosts that haven't sent their logs in?

to4kawa — Fri, 29 May 2020 23:02:47 GMT

host.csv

host
hostA
hostB
hostC
...

yes, it needs just one column host

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Fri, 29 May 2020 23:05:54 GMT

I am going to make this lookup table then figure out how to change my query.

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Fri, 29 May 2020 23:11:48 GMT

also where you stated

("hostA,hostB,hostC ... ",",")

do I place my hosts in there?

Re: How do I make an alert for hosts that haven't sent their logs in?

to4kawa — Fri, 29 May 2020 23:18:52 GMT

 | tstats count where index=abc by host
 | inputlookup append=t host.csv
 | stats sum(count) AS Total BY host | where Total>0

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Fri, 29 May 2020 23:59:52 GMT

Are the hosts in the csv case sensitive?

Re: How do I make an alert for hosts that haven't sent their logs in?

to4kawa — Sat, 30 May 2020 00:06:53 GMT

yes, case sensitive

| tstats count where index=_internal sourcetype="splunkd" by sourcetype 
| append [|makeresults
| eval sourcetype="SplunkD"]
| stats max(count) by sourcetype

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Sat, 30 May 2020 00:20:08 GMT

I am confused now, I don't need to eval the sourcetype I only care about my 3 hosts in my CSV that should be sending data and need it to alert when it doesn't.

Re: How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu — Sat, 30 May 2020 00:29:27 GMT

also I have more than 3 hosts for example I will have another csv file for my linux hosts and windows hosts and then for my high value hosts.

Re: How do I make an alert for hosts that haven't sent their logs in?

to4kawa — Sat, 30 May 2020 00:39:20 GMT

my query on last comment is sample of case sensitive.
You should make the query yourself.

I won't make the query on your behalf.