Scheduled searches no longer running, showing Scheduled Time in the past.

[hanoc]

We have numerous searches that are supposed to run every minute.

They have run successfully for months now, but yesterday we found that they had suddenly stopped doing the Summary Indexing they are supposed to and the scheduled time on the "Manager -> Searches and Reports" page is in the past at the same point as the summary index stops being added to.

If i check in the scheduler.log I can see the search being run with status=success before during and after the time mentioned on the Searches and Reports page.

Any ideas on why this could be happening?

[tnesavich]

I believe you are likely using search head clustering and or pooling and the captain is out of synch. To fix this:

1. Identify your captain: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCdeploymentoverview#Check_search_head...
2. Bounce the captain (Splunk Stop / Start)
3. Confirm your Scheduled searches all have future dates.