Why are skipped alerts appearing?

[bhavneeshvohra]

We have two scheduled alerts :

1) 1st fetch records for last 30 mins and cron expression :29,59 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * *

2) 2nd fetch records for last 6hrs and cron expression:59 5,11,17,23 * * * . .

Here we have found 3 Alerts for 30 mins are skipped, which should be occurred at 10:59 PM, 11:29 PM , 11:59 PM

and 1 Alert for 6 hrs skipped at 11:59 PM.

Why these alerts are getting skipped.?

Please help!!!

[solarboyz1]

index=_internal  sourcetype=scheduler  status="skipped" savedsearch_name="YOUR_SAVED_SEARCH"

You should get events like:

08-25-2019 21:30:02.860 +0000 INFO SavedSplunker - savedsearch_id="nobody;SA-ThreatIntelligence;Threat - Correlation Searches - Lookup Gen", search_type="scheduled", user="admin", app="SA-ThreatIntelligence", savedsearch_name="Threat - Correlation Searches - Lookup Gen", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1, scheduled_time=1566768600, window_time=0

Which should show the reason it was skipped.

[bhavneeshvohra]

Thanks for the reply!!
"The reason we found is maximum number of concurrent search jobs has been reached".
How should w e solve this ? What should be done such that no alerts get skipped anyhow?

[solarboyz1]

If you've reached the maximum number of concurrent search jobs. You need to either increase the limits (if you've got the infrastructure to support it) or reduce the number of searches being run as @snigdhasaxena
describes

[snigdhasaxena]

@bhavneeshvohra raise the priority of your alert, disable the jobs are that running on regular basis and no longer needed or kill the jobs that are already running that have been in queue for long.