topic Re: Alert when host not sending data to splunk in Alerting

Alert when host not sending data to splunk

Sangu — Fri, 28 May 2021 14:51:31 GMT

Hi,

I need to write a query which alerts if any of my host is not sending any logs to splunk in 10mins.

I'm able to get this result.

Challenge here is, I have 2 cluster servers group, if anyone one of the server is sending data then I should exclude the other server from the alert list, not able to write this logic in search.

Example:

Cluster one : Server 1, server 2 and server3

If Server 1 sends data in 10 mins, then I should remove Server 2 and server 3 from the final list.

right now it will only remove server 1 from the list and other 2 servers come in final list.

Like this I have 3 clusters, 2 servers in each cluster.

Please give some ideas on this

Thanks in Advance!

Re: Alert when host not sending data to splunk

richgalloway — Fri, 28 May 2021 15:45:11 GMT

Use an eval command to add a cluster field, setting the value of the cluster based on the name of the server. Then do your counting by cluster rather than by server. Share your current search and we probably can offer more specific suggestions.

Re: Alert when host not sending data to splunk

Sangu — Fri, 28 May 2021 16:55:46 GMT

Hi,

This is the query I used, lookup table has all the servers which needs to send data.

Another details I wanted to add is, not all servers are there in cluster, expect 3 cluster other 10 servers are there which are not part of any cluster.

Re: Alert when host not sending data to splunk

richgalloway — Fri, 28 May 2021 17:01:40 GMT

Perhaps this will help. The case function assigns servers to a cluster. Those that don't belong to a cluster use the host name as the cluster name.

| metasearch index=abc sourcetype=abc | eval host=lower(host) | stats count BY host | append [ inputlookup Devices.csv | eval host=lower(Hostname+".amat.com"), count=0 | fields host count ] | eval cluster=case(host="server1" OR host="server2" OR host="server3","cluster1", host="server4" OR host="server5" OR host="server6", "cluster2", 1==1, host) | stats sum(count) AS total BY cluster | where total=0

Re: Alert when host not sending data to splunk

Sangu — Fri, 28 May 2021 17:26:54 GMT

this worked. Thank you so much for your help!