Why do we have 75,000 skipped scheduled searches over the last 7 days using Splunk 6.0.1?

[dcroteau]

Below are numbers from the search heads which support our Alerting process through Splunk.

So over the last 7 days, We have missed 75,000 schedule searches. I don't understand this because we have less than 250 scheduled searches. We need help to resolve this issue. Please let me know what "best practice action" I should take and what to check to mitigate this skipped searches. All of these searches seem to be running as admin in the Search App.

Each searchhead has 4 core x 6 CPU’s
XXXXXXXm2p had 72,000 skipped searches over the last 7 days
XXXXXXXw2p had 3,600 skipped searches over the last 7 days.

What does Skipped really mean?

[Jarohnimo]

Most of these searches i found are system generated. Perhaps from a Enterprise app that goes haywire with the scheduling. I'm not sure why this isn't better controlled by splunk as out of the box functions are being skipped.

[martin_mueller · Accepted solution]

Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely.

In other words, your scheduler could not run all the searches it was scheduled to run. This can be caused by scheduled searches actually running for much longer than expected, or by other searches blocking too much resources, or by any other resource hog.