Splunk Search

Stats cannot generate alerts?

New Member

I am trying to generate alerts. I have a search query as
index=abc-index host="XYZ123*" collection="AppServer:OrderTracking" counter="Avg. Order Save Time" earliest=-1h
| stats avg(Value) as avgs by host
| where avgs > 5.2

Trying to generate an alert if avgs is larger than 5.2 over the period over last 1 hour. The sample event is like below.

09/01/2017 05:25:19.540 -0700
counter="Avg. Order Save Time"

I have set up the threshold low at this point so that I can test that the alert is generating. When I am searching with this query, getting at least 3 rows in the Statistic tab with 3 host names and avgs > 5.2
I have set up the alert trigger as
Run on Cron Schedule: */5 * * * *
Number of Hosts: is greater than 0 (I have also tried Number of Results also)
Trigger: Once For each result
Throttle: Checked
Suppress triggering for 15 seconds
Trigger Actions: Alert as well as email to my email address.
With this setting I expected alert emails in every 5 minutes, but not receiving none (BTW other alerts with simple search sending alerts). I am not sure whether I am missing any basics. Any suggestion will be highly appreciated.

0 Karma

Esteemed Legend

Take an existing alert that works and clone it. Paste this search's search text into that and change nothing else. Does it work? Probably it will.

0 Karma

New Member

Thanks to mmodestino and woodcock to look into this. Yes, I had put */5 * * * * for cron. It showed something like "no event/trigger fired" (though there were a few rows). After spending several hours, i deleted the alert and added as a new from scratch. It worked! Unfortunately, I could not find the error in the previous set up or search. Thanks for your suggestion on throttle.

0 Karma

Esteemed Legend

Be sure to click Accept to close the question.

0 Karma

Splunk Employee
Splunk Employee

Hi dban2005!

Can I assume you meant */5 * * * * is your cron? So running every 5 mins looking back an hour?

What happens if set it with the action "List in Triggered Alerts"?

I like to use that as the test before relying on email or other means of sending communication of the alert.

Also try setting the time range in the alert config rather than in your search. My hunch is that your scheduled search the alert is using has something set incorrectly.

Also your throttle is unnecessary as it will only fire every 5 mins....(assuming I have ur cron right).

alt text

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...