Alerting

How to create a search to check flatline for metrics?

sowji589
New Member

I am actually trying to trigger an alert when Splunk is not receiving the metrics. For now, I am checking if the value is 0 trigger an alert but I am not sure if I am doing it correct. Can someone help me in this regard? Thanks in advance.

Labels (1)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

If you know the exact origin of the data (by example host, sourcetype, fields), then you can have a search that look for that data over a recent timerange, (stats count) and trigger if there are no results at all.

 

But if you are actually searching dynamically over several origins  (| stats count by host). Then you need to compare to a list, or older data to notice that once origin is missing.

- It could be am hardcoded count, a lookup you maintain, it could be a meta data search, it could be a subsearch with a different timerange....

- or I could be a search that is looking back on a longer timerange, and do a ( | timechart count by host) Or ( | bucket _time span=1h | stats count by _time host), and has some logic to check if the recent intervals are zero). But this may require more longer/expensive searches each time, so it's not good for a frequent alert. 

0 Karma

gjanders
SplunkTrust
SplunkTrust

TrackMe might also work here as an app to monitor all data hosts, it includes metrics.

BrokenHosts and Metawoot! may also do metrics in the future.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...