Alerting

Alert when host not sending data to splunk

Sangu
Explorer

Hi,

I need to write a query which alerts if any of my host is not sending any logs to splunk in 10mins.

I'm able to get this result.

Challenge here is, I have 2 cluster servers group, if anyone one of the server is sending data then I should exclude the other server from the alert list, not able to write this logic in search.

Example:

Cluster one :  Server 1, server 2 and server3 

If Server 1 sends data in 10 mins, then I should remove Server 2 and server 3 from the final list.

right now it will only remove server 1 from the list and other 2 servers come in final list.

Like this I have 3 clusters, 2 servers in each cluster.

Please give some ideas on this

Thanks in Advance!

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Perhaps this will help.  The case function assigns servers to a cluster.  Those that don't belong to a cluster use the host name as the cluster name.

| metasearch index=abc sourcetype=abc
| eval host=lower(host)
| stats count BY host
| append
[ inputlookup Devices.csv
| eval host=lower(Hostname+".amat.com"), count=0
| fields host count ]
| eval cluster=case(host="server1" OR host="server2" OR host="server3","cluster1", host="server4" OR host="server5" OR host="server6", "cluster2", 1==1, host)
| stats sum(count) AS total BY cluster | where total=0

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Use an eval command to add a cluster field, setting the value of the cluster based on the name of the server.  Then do your counting by cluster rather than by server.  Share your current search and we probably can offer more specific suggestions.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Sangu
Explorer

Hi,

| metasearch index=abc sourcetype=abc
| eval host=lower(host)
| stats count BY host
| append
[ inputlookup Devices.csv
| eval host=lower(Hostname+".amat.com"), count=0
| fields host count ]
| stats sum(count) AS total BY host | where total=0

 

This is the query I used, lookup table has all the servers which needs to send data. 

Another details I wanted to add is, not all servers are there in cluster, expect 3 cluster other 10 servers are there which are not part of any cluster.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Perhaps this will help.  The case function assigns servers to a cluster.  Those that don't belong to a cluster use the host name as the cluster name.

| metasearch index=abc sourcetype=abc
| eval host=lower(host)
| stats count BY host
| append
[ inputlookup Devices.csv
| eval host=lower(Hostname+".amat.com"), count=0
| fields host count ]
| eval cluster=case(host="server1" OR host="server2" OR host="server3","cluster1", host="server4" OR host="server5" OR host="server6", "cluster2", 1==1, host)
| stats sum(count) AS total BY cluster | where total=0

 

---
If this reply helps you, Karma would be appreciated.

Sangu
Explorer

this worked. Thank you so much for your help!

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...