Health monitoring

[yav2810]

The system sends the information (the value of the counter) about the number of events transmitted during this time interval approximately in 5 minutes. It is necessary to build a monitoring system - to display information about sources that do not transmit events. The task is divided into 2 blocks: 1 - the source is available, but the counter value is zero, 2 - the source is not available (events do not arrive at Splunk at all).

The first task is easily solved. It builds statistics on the number of devices that transmitted events in the last hour, but the counter was zero:

query
| stats sum(cn2) as tCount by shost
| where tCount=0
| stats count(shost)

cn2 - counter

Outputting a list of sources that pass the counter value to zero:

query
| stats sum(cn2) as tCount by shost
| where tCount=0
| fields shost

The second task is more complicated. I consider the number of devices from which messages were received in the last hour:

query
| dedup shost
| stats count(shost) as tCount

It remains to solve the problem of obtaining a list of devices that did not wait for events in the last hour (were not available). I ask to help with the decision of the given task

[HiroshiSatoh · Accepted solution]

For example, you can search by this search sentence.

|tstats count by host|search NOT [search index=* earliest=-1h|dedup host|table host]
OR
|inputlookup host_list.csv|search NOT [search index=* earliest=-1h|dedup host|table host]

[yav2810]

Thank you