How to detect logs missing from some sources

[aiumjw1]

Hi Team,

I have logs coming from certain nodes and clusters. How can I detect if the logs go missing even from one of the clusters. The nodes and clusters are under the field name source. 

For example, I have source = logs/node*c*
node* has 3 to 4 nodes.
c* have 8 to 10 clusters.

I want to create an alert to notify if logs are missing even from one cluster.

Thanks.

[ITWhisperer]

A classic question! Splunk is good at easily finding things that are there, not so good at easily finding things which aren't there. Basically, you need to tell Splunk what to look for and see if it finds any. For example:

<your search>
| stats count by source
| append
  [| makeresults
  | fields - _time
  | eval source=split("source1,source2,source3",",")
  | mvexpand source
  | eval count = 0]
| stats sum(count) as count by source
| where count == 0

[aiumjw1]

Thank you. This helps. But I have a question. Above query tells me the sources list and number of logs coming in Splunk from which I can filter the source having zero logs. But let's take below example,

If for 1hour I am not getting logs from a particular source, then I think in the above query it will not be listed at all. How can I find that one source which is missing.

[ITWhisperer]

If you include the "missing" source in the list of sources that are appended with the makeresults, why do you think it wouldn't appear in the list?