Alerting

Alert stopped triggering

ckeller2791
Explorer

Have an alert that use to trigger and send an email, the alert details are

SEARCH STRING: source="WinEventLog:Security" EventCode=4740
| eval SecurityID = mvfilter(match(Security_ID,"DOMAIN*"))
| eval Computer = Caller_Computer_Name
| dedup SecurityID,_time
| table _time,SecurityID,Computer

Alert Type :Scheduled , Run on Cron Schedule
Time Range: Last 15 minutes
Cron Expression: */5 * * * *
Trigger alert when: Number of Results, is greater than 0
Trigger: For each result
Trigger Actions: Add to Triggered Alerts, Send email

On Friday last week it was working, come Monday the alert does not trigger. I've tested the search and even added to a dashboard and I can see the output in the dashboard. However the alert is never triggered, it no longer shows in the "Triggered Alerts". I've even ran the search and added the "sendemail" command which was successful. I've altered the permission on the alert to "Everyone" read/write and still has not affect. Some direction or suggestions would be welcomed.

-Chris Keller

Tags (1)
0 Karma
1 Solution

mayurr98
Super Champion

did you check internal logs for the errors?
https://answers.splunk.com/answers/517/how-to-search-recent-alerts-fired-by-splunk.html

index=_internal sourcetype=scheduler

View solution in original post

0 Karma

mayurr98
Super Champion

did you check internal logs for the errors?
https://answers.splunk.com/answers/517/how-to-search-recent-alerts-fired-by-splunk.html

index=_internal sourcetype=scheduler

0 Karma

ckeller2791
Explorer

Thanks, found the problem even though doesn't show as an error:

03-05-2019 16:20:00.662 -0500 INFO SavedSplunker - savedsearch_id="nobody;search;Active Directory Accounts Locked Out", search_type="scheduled", user="admin", app="search", savedsearch_name="Active Directory Accounts Locked Out", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide"

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...

Monitoring AI Agents with Splunk Observability Cloud

Let’s say I’m running a travel planning AI app in production. A user asks for three concise hotel options in ...

[Puzzles] Solve, Learn, Repeat: Tiling

This puzzle (first published here) is based on finding groups of tessellated tiles (inspired by floor tiles I ...