Alert if there are no logs (by host and by sourcetype)

[woodentree]

Hello,

We scheduled a search that alerts us if we do not receive logs from any of our hosts since >5 minutes. It looks like below:

| metadata type=hosts index=* | eval age=now()-lastTime | where age>3600

However, there is an issue - it does not work if we partly receive logs from the host (let's say, only 1 sourcetype out of 2).

Do you know a way to create the same alert by host AND by sourcetype at the same time?

Thanks for the help.

[sumanssah · Accepted solution]

Try this

| tstats latest(_time) as lastSeen where index IN(*) by host sourcetype
|eval delay=round ((now() - lastSeen)/60/60/24,2)
|eval lastSeen=strftime(lastSeen, "%Y/%m/%d %H:%M:%S")
|search delay>1 
|eval HostName=lower(host)

[woodentree]

Gorgeous! That what we was searching for.
Thanks for the help!

[sumanssah]

Thanks for confirming 🙂

[gcusello]

Hi @woodentree,
you have to create a lookup containing the list of hosts to monitor (called e.g. perimeter.csv) containing at least one field (host) or also more information.
Then you can schedule a search e.g. every five minutes like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

In this case you're sure that an host is sending logs to Splunk.
If instead you want to monitor that are arriving logs on an index with a sourcetype, you can modify the main search in this way, using the same approach:

| metasearch index=your_index sourcetype=your_sourcetype
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Ciao.
Giuseppe

[woodentree]

Hi @gcusello,

Appreciate your help!

However, our goal is slightly different: we do not like to monitor some particular sourcetype, but all of them. The goal is to know if any of our hosts does not receive logs of any sourcetype.

Thanks.

[gcusello]

your use case is simplar than I thought: you need to know if there are hosts that don't receive anything, so try something like this:

 | metasearch index=*
 | eval host=lower(host)
 | stats count BY host
 | append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
 | stats sum(count) AS total BY host
 | where total=0

Ciao.
Giuseppe

[woodentree]

Got it.
Thank you @gcusello !