Alerting

Why are skipped alerts appearing?

bhavneeshvohra
Engager

We have two scheduled alerts :

1) 1st fetch records for last 30 mins and cron expression :29,59 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * *

2) 2nd fetch records for last 6hrs and cron expression:59 5,11,17,23 * * * . .

Here we have found 3 Alerts for 30 mins are skipped, which should be occurred at 10:59 PM, 11:29 PM , 11:59 PM

and 1 Alert for 6 hrs skipped at 11:59 PM.

Why these alerts are getting skipped.?

Please help!!!

0 Karma

solarboyz1
Builder
index=_internal  sourcetype=scheduler  status="skipped" savedsearch_name="YOUR_SAVED_SEARCH"

You should get events like:

08-25-2019 21:30:02.860 +0000 INFO SavedSplunker - savedsearch_id="nobody;SA-ThreatIntelligence;Threat - Correlation Searches - Lookup Gen", search_type="scheduled", user="admin", app="SA-ThreatIntelligence", savedsearch_name="Threat - Correlation Searches - Lookup Gen", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1, scheduled_time=1566768600, window_time=0

Which should show the reason it was skipped.

0 Karma

bhavneeshvohra
Engager

Thanks for the reply!!
"The reason we found is maximum number of concurrent search jobs has been reached".
How should w e solve this ? What should be done such that no alerts get skipped anyhow?

0 Karma

solarboyz1
Builder

If you've reached the maximum number of concurrent search jobs. You need to either increase the limits (if you've got the infrastructure to support it) or reduce the number of searches being run as @snigdhasaxena
describes

0 Karma

snigdhasaxena
Communicator

@bhavneeshvohra raise the priority of your alert, disable the jobs are that running on regular basis and no longer needed or kill the jobs that are already running that have been in queue for long.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...