topic Re: Alert if there are no logs (by host and by sourcetype) in Splunk Search

Alert if there are no logs (by host and by sourcetype)

woodentree — Tue, 25 Feb 2020 16:19:24 GMT

Hello,

We scheduled a search that alerts us if we do not receive logs from any of our hosts since >5 minutes. It looks like below:

| metadata type=hosts index=* | eval age=now()-lastTime | where age>3600

However, there is an issue - it does not work if we partly receive logs from the host (let's say, only 1 sourcetype out of 2).

Do you know a way to create the same alert by host AND by sourcetype at the same time?

Thanks for the help.

Re: Alert if there are no logs (by host and by sourcetype)

gcusello — Tue, 25 Feb 2020 16:57:22 GMT

Hi @woodentree,
you have to create a lookup containing the list of hosts to monitor (called e.g. perimeter.csv) containing at least one field (host) or also more information.
Then you can schedule a search e.g. every five minutes like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

In this case you're sure that an host is sending logs to Splunk.
If instead you want to monitor that are arriving logs on an index with a sourcetype, you can modify the main search in this way, using the same approach:

| metasearch index=your_index sourcetype=your_sourcetype
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Ciao.
Giuseppe

Re: Alert if there are no logs (by host and by sourcetype)

woodentree — Thu, 27 Feb 2020 13:23:41 GMT

Hi @gcusello,

Appreciate your help!

However, our goal is slightly different: we do not like to monitor some particular sourcetype, but all of them. The goal is to know if any of our hosts does not receive logs of any sourcetype.

Thanks.

Re: Alert if there are no logs (by host and by sourcetype)

gcusello — Thu, 27 Feb 2020 13:34:10 GMT

your use case is simplar than I thought: you need to know if there are hosts that don't receive anything, so try something like this:

 | metasearch index=*
 | eval host=lower(host)
 | stats count BY host
 | append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
 | stats sum(count) AS total BY host
 | where total=0

Ciao.
Giuseppe

Re: Alert if there are no logs (by host and by sourcetype)

sumanssah — Thu, 27 Feb 2020 13:49:38 GMT

Try this

| tstats latest(_time) as lastSeen where index IN(*) by host sourcetype
|eval delay=round ((now() - lastSeen)/60/60/24,2)
|eval lastSeen=strftime(lastSeen, "%Y/%m/%d %H:%M:%S")
|search delay>1 
|eval HostName=lower(host)

Re: Alert if there are no logs (by host and by sourcetype)

woodentree — Thu, 27 Feb 2020 14:15:38 GMT

Gorgeous! That what we was searching for.
Thanks for the help!

Re: Alert if there are no logs (by host and by sourcetype)

sumanssah — Thu, 27 Feb 2020 15:24:49 GMT

Thanks for confirming 🙂

Re: Alert if there are no logs (by host and by sourcetype)

woodentree — Fri, 28 Feb 2020 08:22:41 GMT

Got it.
Thank you @gcusello !