What are the main reasons for scheduled search latency?

[pradeepkumarg]

The difference between the dispatched_time and scheduled_time on the scheduler logs--What is this exactly telling? What are the reasons for this delay?

Could it be because the cluster captain is busy? Is there a limit on how many schedule searches can the captain delegate ?
Is it because search head members are exhausted?
Is it because the indexers are busy? Does the search head members even have visibility into the indexer performance and CPU availability?

Any insight into this is greatly appreciated. Thank you.

Regards,
Pradeep

[lguinn2]

Whether you have a search head cluster or independent search heads, the "scheduler" is responsible for determining when reports run, and if necessary, which reports to skip. Here is some key documentation on the scheduler
Configure the priority of scheduled reports

The biggest reasons that a scheduled search fails to run is that the search head has too many simultaneous jobs to run. The documentation has a good description of how that happens and what you can do in the scheduler to manage it.

In a distributed environment, a search needs resources on both the search head and the indexer tier in order to execute. As a rule of thumb, search performance can be improved by adding more indexers, since indexers do the actual searching. While adding indexers does not increase the number of simultaneous searches, it does increase the speed of the searches so that more searches execute within a given time.

HTH