Splunk Ent. v.6.5.2
I set up a few alerts to run every 5min with condition if # of events > 0.
I know for a fact that the search will return > 0 because I set up my time range for a few hours where it always returns > 0.
Search query:
index=index001 Source="Record Alert"
| stats count(eval(like(Description,"%orders failed to record on at least%"))) AS Occurences
Any ideas where to start troubleshooting this? I dont see anything on Activity-->Triggered Alerts
Another thing that is weird when I run the query below I get ZERO results for ALL TIME:
index=_internal log_level=warn* OR log_level=err*
EDIT: I just looked at the scheduler.log and it shows: status=success, digest_mode=1 for my alert but I dont think it is triggering at all. It stills shows ZERO for the "Alerts" field under the "Searches, reports, and alerts" interface.
Thank you
I had similar issue
Try to search your alerts in skipped search using savedsearch_id
index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped
We had to change parameters in limits.conf
I had similar issue
Try to search your alerts in skipped search using savedsearch_id
index=_internal earliest=[your_time] sourcetype=scheduler search_type=scheduled status=skipped
We had to change parameters in limits.conf
thanks. that helped me understand the logs. they were all SUCCESS. I was relying on the Splunk UI and it was not showing me anything Under the Alerts. it was always ZERO. Go figure.
The first query is not appropriate .
try this
index=index001 Source="Record Alert"
Description="*orders failed to record *"| stats count AS Occurences
For the secondd try this
index=_internal log_level=WARN* OR log_level=err* OR log_leval=ERROR*
why the first query is not appropriate? your query will only capture an exact string right?
and the second query didnt work either. I get ZERO back. It is in our lab but the query works in Production. Not sure why.
why the first query is not appropriate? your query will only capture an exact string right?
Yeah
your query will return ZERO on my search. I still don't understand why my query is wrong. It returns 95 events average. And the questions remains, why the alarm won't trigger????
what alert action did you opt for ?