Alerting

Alert stopped triggering

ckeller2791
Explorer

Have an alert that use to trigger and send an email, the alert details are

SEARCH STRING: source="WinEventLog:Security" EventCode=4740
| eval SecurityID = mvfilter(match(Security_ID,"DOMAIN*"))
| eval Computer = Caller_Computer_Name
| dedup SecurityID,_time
| table _time,SecurityID,Computer

Alert Type :Scheduled , Run on Cron Schedule
Time Range: Last 15 minutes
Cron Expression: */5 * * * *
Trigger alert when: Number of Results, is greater than 0
Trigger: For each result
Trigger Actions: Add to Triggered Alerts, Send email

On Friday last week it was working, come Monday the alert does not trigger. I've tested the search and even added to a dashboard and I can see the output in the dashboard. However the alert is never triggered, it no longer shows in the "Triggered Alerts". I've even ran the search and added the "sendemail" command which was successful. I've altered the permission on the alert to "Everyone" read/write and still has not affect. Some direction or suggestions would be welcomed.

-Chris Keller

Tags (1)
0 Karma
1 Solution

mayurr98
Super Champion

did you check internal logs for the errors?
https://answers.splunk.com/answers/517/how-to-search-recent-alerts-fired-by-splunk.html

index=_internal sourcetype=scheduler

View solution in original post

0 Karma

mayurr98
Super Champion

did you check internal logs for the errors?
https://answers.splunk.com/answers/517/how-to-search-recent-alerts-fired-by-splunk.html

index=_internal sourcetype=scheduler

0 Karma

ckeller2791
Explorer

Thanks, found the problem even though doesn't show as an error:

03-05-2019 16:20:00.662 -0500 INFO SavedSplunker - savedsearch_id="nobody;search;Active Directory Accounts Locked Out", search_type="scheduled", user="admin", app="search", savedsearch_name="Active Directory Accounts Locked Out", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide"

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Splunk Enterprise Security: An Automated Phishing Response Workflow

Originally posted on Splunk.com.   Key takeaways  The volume of phishing emails can overwhelm even the most ...

Sharper ES Detections, Agentic AI Integrations, Smarter Workload Management, and More ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

Data Outage Monitoring and Reporting

We’ve all been there, right? Living our best life, working on our Splunk environment, when out of nowhere a ...