Alerting

How to create a search to check flatline for metrics?

sowji589
New Member

I am actually trying to trigger an alert when Splunk is not receiving the metrics. For now, I am checking if the value is 0 trigger an alert but I am not sure if I am doing it correct. Can someone help me in this regard? Thanks in advance.

Labels (1)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

If you know the exact origin of the data (by example host, sourcetype, fields), then you can have a search that look for that data over a recent timerange, (stats count) and trigger if there are no results at all.

 

But if you are actually searching dynamically over several origins  (| stats count by host). Then you need to compare to a list, or older data to notice that once origin is missing.

- It could be am hardcoded count, a lookup you maintain, it could be a meta data search, it could be a subsearch with a different timerange....

- or I could be a search that is looking back on a longer timerange, and do a ( | timechart count by host) Or ( | bucket _time span=1h | stats count by _time host), and has some logic to check if the recent intervals are zero). But this may require more longer/expensive searches each time, so it's not good for a frequent alert. 

0 Karma

gjanders
SplunkTrust
SplunkTrust

TrackMe might also work here as an app to monitor all data hosts, it includes metrics.

BrokenHosts and Metawoot! may also do metrics in the future.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...