Why is my simple alert not being triggered with the condition "Number of results > 500 in 4 hours"?

[razlani]

Hi all,

Just setting up alerts for the first time and I've selected this as search string:

index=blah sourcetype=error | stats count as amount

I've also tried:

index=subsites sourcetype=apache_error

What I wish to do is "Email me when events for this search total an amount > 500 for the past 4 hour window, then wait 4 hours before checking again."

The search itself returns 10's of thousands of events so I know I'm good there. I change time period to last 4 hours. I then do save as > alert as instructed in the docs.

Alert type: REAL TIME

Trigger Condition: Number of results > 500 in 4 Hours

List in Triggered Alerts - YES

Send Email - YES

For now I've left throttle off as I don't care if I get spammed - I just want it to work. When I check the alerts list I see it's not triggering the alert - when I "view recent" on the alert I see it has 0 events (or something like 2 if I use the "| stats count as amount" within the search string.

It's entirely possible (likely) I've misunderstood the "rolling window" search or the criteria by which it triggers the alert (not sure if I have to use stats here, or allow it to count the events for me when I create alert for example) - Please help!!!!

[somesoni2 · Accepted solution]

If you're looking through the data that was indexed in Splunk in Last few hours, it's a historical data and you don't need to run realtime searches for the alerts. Here is what I would do (for your first requirement in the question)

Search : index=subsites sourcetype=apache_error
Start time: -4h@h
Finish time: @h

Schedule type: Cron
Cron schedule :  2 */4 * * *    (this will ensure to run the alert search every 4 hours)

Alert 
Condition = If number of events is greater than 500

***All remaining alert action what you're doing right now

[razlani]

Have my babies sir, have them all.

[razlani]

Ok so let's be very clear:

If I type:

sourcetype="mysqld" NOT "[NOTE]"

In the search app I get one event with date 20/03/2015 and time 12:00 PM.

My question simplified is this - how can I capture that event with an alert? I've tried the above search and changed the time range to:

Start time: rt-30h
End time: rt-0h

Number of events > 0, alert mode once per search, as per the following screenshot: http://i.imgur.com/T07lrT0.png

I've got the alerts working for windows where the events are occurring in real time in the present, but in order to test other alerts I'd want to start by capturing past events as per above.

Please help!