How to set up a one time real-time alert only when a value exceeds a threshold?

[melonman]

Hi,

I am trying to set up an alert with the following events.
The alert condition is one time alert only when a value exceeds a threshold.

If threshold is 5, then I want to get alerts at the time with bold event below.

2014/09/24 09:41:58 +0100 val=3
2014/09/24 09:41:59 +0100 val=3
2014/09/24 09:42:00 +0100 val=3
2014/09/24 09:42:01 +0100 val=9  <-- alert here!
2014/09/24 09:42:02 +0100 val=9
2014/09/24 09:42:03 +0100 val=9
2014/09/24 09:42:04 +0100 val=9
2014/09/24 09:42:05 +0100 val=9
2014/09/24 09:42:06 +0100 val=9
2014/09/24 09:42:07 +0100 val=9
2014/09/24 09:42:08 +0100 val=3
2014/09/24 09:42:09 +0100 val=3
2014/09/24 09:42:10 +0100 val=3
2014/09/24 09:42:11 +0100 val=9   <-- alert here!
2014/09/24 09:42:12 +0100 val=9
2014/09/24 09:42:13 +0100 val=9
2014/09/24 09:42:14 +0100 val=9
2014/09/24 09:42:15 +0100 val=3
2014/09/24 09:42:16 +0100 val=3
2014/09/24 09:42:17 +0100 val=9   <-- alert here!
2014/09/24 09:42:18 +0100 val=9
2014/09/24 09:42:19 +0100 val=9
2014/09/24 09:42:20 +0100 val=9
2014/09/24 09:42:21 +0100 val=9

I want it to be real-time.

Would you please give me a comment on this?
Thank you in advance.

[ltrand · Accepted solution]

Here is going to be the issue, to be real time this search needs to schedule, execute, and complete every second for it to work as a real-time alert. This is not the ideal use for Splunk. If you can tolerate a 15 minute window then it could potentially work depending on the environment.

To get realtime alerting I would suggest an SNMP engine. It's built for traps and polling. SolarWinds, OpenNMI, or NAGIOS, they are all excellent at doing real-time alerting.

[a212830]

15 minutes? That seems a bit excessive - you should be able to schedule a query that runs and looks for this information as needed. Real-time will depend on how many other real-time searches you have going (and capacity, performance...), but you can also do things like every 1 minute, 2 minutes....