Alert when host not sending data to splunk

[Sangu]

Hi,

I need to write a query which alerts if any of my host is not sending any logs to splunk in 10mins.

I'm able to get this result.

Challenge here is, I have 2 cluster servers group, if anyone one of the server is sending data then I should exclude the other server from the alert list, not able to write this logic in search.

Example:

Cluster one :  Server 1, server 2 and server3 

If Server 1 sends data in 10 mins, then I should remove Server 2 and server 3 from the final list.

right now it will only remove server 1 from the list and other 2 servers come in final list.

Like this I have 3 clusters, 2 servers in each cluster.

Please give some ideas on this

Thanks in Advance!

[richgalloway]

Use an eval command to add a cluster field, setting the value of the cluster based on the name of the server.  Then do your counting by cluster rather than by server.  Share your current search and we probably can offer more specific suggestions.

[Sangu]

Hi,

| metasearch index=abc sourcetype=abc
| eval host=lower(host)
| stats count BY host
| append
[ inputlookup Devices.csv
| eval host=lower(Hostname+".amat.com"), count=0
| fields host count ]
| stats sum(count) AS total BY host | where total=0

This is the query I used, lookup table has all the servers which needs to send data. 

Another details I wanted to add is, not all servers are there in cluster, expect 3 cluster other 10 servers are there which are not part of any cluster.

[richgalloway · Accepted solution]

Perhaps this will help.  The case function assigns servers to a cluster.  Those that don't belong to a cluster use the host name as the cluster name.

| metasearch index=abc sourcetype=abc
| eval host=lower(host)
| stats count BY host
| append
[ inputlookup Devices.csv
| eval host=lower(Hostname+".amat.com"), count=0
| fields host count ]
| eval cluster=case(host="server1" OR host="server2" OR host="server3","cluster1", host="server4" OR host="server5" OR host="server6", "cluster2", 1==1, host)
| stats sum(count) AS total BY cluster | where total=0

[Sangu]

this worked. Thank you so much for your help!