Alerting

How do I make an alert for hosts that haven't sent their logs in?

splunktrainingu
Communicator

I am only curious for a certain index

index=abc
| stats count by host
| stats sum(count) AS Total BY host
| where Total>0

This search is good to see how many logs are coming in for my hosts in that index but the problem is when a host stops sending I have no alert for it. I tried changing the "|where Total>=0" but it took off the host from my table when it hit zero. How can I adjust or change my query to make it so I can alert when a host hits 0 logs.

Tags (1)
0 Karma

gjanders
SplunkTrust
SplunkTrust

If you instead are looking for a more global solution consider:
TrackMe
Or Meta Woot!
Or Broken hosts

I use TrackMe at the moment 🙂

0 Karma

splunktrainingu
Communicator

Thank you, my concern is does this app need to go on my forwarders or the is the app downloaded to app folder on my splunk indexer?

0 Karma

to4kawa
Ultra Champion
index=abc 
| stats count by host 
| append [| makeresults
| eval host=split("hostA,hostB,hostC ... ",",")
| mvexpand host
| fields host
| table host]
| stats sum(count) AS Total BY host | where Total>0

instead of append [....] , you can create host.csv and use | inputlookup append=t host.csv

0 Karma

splunktrainingu
Communicator

using this lookup table would it just have one column that being my hosts and the servers underneath it with no other columns?

0 Karma

to4kawa
Ultra Champion

host.csv

host
hostA
hostB
hostC
...

yes, it needs just one column host

0 Karma

splunktrainingu
Communicator

also where you stated

("hostA,hostB,hostC ... ",",")

do I place my hosts in there?

0 Karma

to4kawa
Ultra Champion
 | tstats count where index=abc by host
 | inputlookup append=t host.csv
 | stats sum(count) AS Total BY host | where Total>0
0 Karma

splunktrainingu
Communicator

Are the hosts in the csv case sensitive?

0 Karma

to4kawa
Ultra Champion

yes, case sensitive

| tstats count where index=_internal sourcetype="splunkd" by sourcetype 
| append [|makeresults
| eval sourcetype="SplunkD"]
| stats max(count) by sourcetype
0 Karma

splunktrainingu
Communicator

I am confused now, I don't need to eval the sourcetype I only care about my 3 hosts in my CSV that should be sending data and need it to alert when it doesn't.

0 Karma

to4kawa
Ultra Champion

my query on last comment is sample of case sensitive.
You should make the query yourself.

I won't make the query on your behalf.

0 Karma

splunktrainingu
Communicator

also I have more than 3 hosts for example I will have another csv file for my linux hosts and windows hosts and then for my high value hosts.

0 Karma

splunktrainingu
Communicator

I am going to make this lookup table then figure out how to change my query.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...