Schedule window vs scheduling mode

[peterschloenske]

Hi,

I want to prevent alerts from being skipped and I'm fine, that the alerts don't run at a specific time. I prefer to be notified with a delay than not at all. 

One option is to set a schedule window. First of all, I'm wondering why the Alert Editing does not offer this option like reports do. I have to navigate to the Advanced Edit Mode to configure the schedule window. When it is configured, we allow the scheduler to delay the dispatch time. But at some point the search will be skipped anyway.

Another option is to use the scheduling mode "continuous".  As far as I understand it, an alert with mode "continuous" is never skipped, which sounds reasonable to have a security monitoring without gaps.  I assume the scheduler will try to run the search as soon as possible.

• Is the continuous mode a best practice to avoid gaps or are there valid reasons not to use it? If the mode is used it might be a good idea to observe the scheduler lag more closely to determine "how late" alerts run and if the scheduler is building a huge backlog of delayed searches.
  
• I don't know how the scheduling_mode interacts with the schedule window. Does the schedule window have any effect, when the mode is "continuous"?
  
  

[smurf]

Hi,

As I understand it. 

Continuous searches are never skipped and will be run whenever Splunk is available after downtime or when it has the resources to run it. The downside is that real-time searches have higher priority, so if your pipeline is filled with real-time searches, your continuous search might never run. Or so I was told. I never had an issue with it when I used it, but our partner suggested migrating to real-time searches. 

After that, we used real-time searches for almost anything while specifying a larger search window with matching throttling.

I suggest going through these articles as they might answer most of your questions:

Prioritize concurrently scheduled reports in Splunk Web

Configure the priority of scheduled reports (real-time vs. continuous scheduling)

smurf

[isoutamo]

Quite interesting advice! Real time search/alert reserve one core from all search peers. This means that you couldn’t run more than core amount of individual search peer - 3-4 which are used for ingesting and running Splunk’s other core services. For long run this leads situation where you run out of resources and you cannot use splunk for anything else!

Actually I haven’t been a situation when I have had to run real time alert. Usually there are way to use scheduled alert instead of real time.

r. Ismo

[smurf]

I don't mean real-time searches but real-time schedule type.
That's the type of schedule that would skip time windows, unlike continuous schedule which would continue where it left of. That's why I used longer search windows, so if a few runs are skipped, I would still query all logs from the downtime period. 

The name is very confusing, TBH.