Alerting

Schedule Alerts not being triggered

maximusdm
Communicator

Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.

Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences

Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts

Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*

EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.

Thank you
alt text

Tags (1)
0 Karma
1 Solution

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

View solution in original post

0 Karma

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

0 Karma

maximusdm
Communicator

thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.

0 Karma

naidusadanala
Communicator

The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences

For the secondd try this

index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*

0 Karma

maximusdm
Communicator

why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.

0 Karma

naidusadanala
Communicator

why the first query is not appropriate? your query will only capture an exact string right?

Yeah

0 Karma

maximusdm
Communicator

your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????

0 Karma

naidusadanala
Communicator

what alert action did you opt for ?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...