Schedule Alerts not being triggered

[maximusdm]

Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.

Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences

Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts

Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*

EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.

Thank you

[andrey2007 · Accepted solution]

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

[maximusdm]

thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.

[naidusadanala]

The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences

For the secondd try this

index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*

[maximusdm]

why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.

[naidusadanala]

why the first query is not appropriate? your query will only capture an exact string right?

Yeah

[maximusdm]

your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????

[naidusadanala]

what alert action did you opt for ?