Alerting

Schedule Alerts not being triggered

maximusdm
Communicator

Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.

Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences

Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts

Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*

EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.

Thank you
alt text

Tags (1)
0 Karma
1 Solution

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

View solution in original post

0 Karma

andrey2007
Contributor

I had similar issue
Try to search your alerts in skipped search using savedsearch_id

index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped

We had to change parameters in limits.conf

0 Karma

maximusdm
Communicator

thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.

0 Karma

naidusadanala
Communicator

The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences

For the secondd try this

index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*

0 Karma

maximusdm
Communicator

why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.

0 Karma

naidusadanala
Communicator

why the first query is not appropriate? your query will only capture an exact string right?

Yeah

0 Karma

maximusdm
Communicator

your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????

0 Karma

naidusadanala
Communicator

what alert action did you opt for ?

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...