Alerting

How to create a search to check flatline for metrics?

sowji589
New Member

I am actually trying to trigger an alert when Splunk is not receiving the metrics. For now, I am checking if the value is 0 trigger an alert but I am not sure if I am doing it correct. Can someone help me in this regard? Thanks in advance.

Labels (1)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

If you know the exact origin of the data (by example host, sourcetype, fields), then you can have a search that look for that data over a recent timerange, (stats count) and trigger if there are no results at all.

 

But if you are actually searching dynamically over several origins  (| stats count by host). Then you need to compare to a list, or older data to notice that once origin is missing.

- It could be am hardcoded count, a lookup you maintain, it could be a meta data search, it could be a subsearch with a different timerange....

- or I could be a search that is looking back on a longer timerange, and do a ( | timechart count by host) Or ( | bucket _time span=1h | stats count by _time host), and has some logic to check if the recent intervals are zero). But this may require more longer/expensive searches each time, so it's not good for a frequent alert. 

0 Karma

gjanders
SplunkTrust
SplunkTrust

TrackMe might also work here as an app to monitor all data hosts, it includes metrics.

BrokenHosts and Metawoot! may also do metrics in the future.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...