Alerting

Schedule Alerts not being triggered

maximusdm
Communicator

Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.

Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences

Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts

Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*

EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.

Thank you
alt text

Tags (1)
0 Karma
1 Solution

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

View solution in original post

0 Karma

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

0 Karma

maximusdm
Communicator

thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.

0 Karma

naidusadanala
Communicator

The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences

For the secondd try this

index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*

0 Karma

maximusdm
Communicator

why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.

0 Karma

naidusadanala
Communicator

why the first query is not appropriate? your query will only capture an exact string right?

Yeah

0 Karma

maximusdm
Communicator

your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????

0 Karma

naidusadanala
Communicator

what alert action did you opt for ?

0 Karma
Get Updates on the Splunk Community!

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...