topic Re: Schedule Alerts not being triggered in Alerting

Schedule Alerts not being triggered

maximusdm — Tue, 29 Sep 2020 14:00:40 GMT

Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.

Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences

Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts

Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*

EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.

Thank you

Re: Schedule Alerts not being triggered

naidusadanala — Tue, 29 Sep 2020 14:00:38 GMT

The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences

For the secondd try this

index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*

Re: Schedule Alerts not being triggered

maximusdm — Wed, 10 May 2017 15:18:12 GMT

why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.

Re: Schedule Alerts not being triggered

naidusadanala — Wed, 10 May 2017 15:42:34 GMT

why the first query is not appropriate? your query will only capture an exact string right?

Yeah

Re: Schedule Alerts not being triggered

maximusdm — Wed, 10 May 2017 16:13:10 GMT

your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????

Re: Schedule Alerts not being triggered

naidusadanala — Wed, 10 May 2017 19:18:15 GMT

what alert action did you opt for ?

Re: Schedule Alerts not being triggered

andrey2007 — Tue, 29 Sep 2020 14:01:23 GMT

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

Re: Schedule Alerts not being triggered

maximusdm — Thu, 11 May 2017 21:43:23 GMT

thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.