I have a splunk search head that has a bunches of scheduled searches which runs every minute or so. However, since after 5/15, not one scheduled search has ran, and no alert email has been received since.
I looked at the scheduler.log, and it seems to have confirmed that, as there were no new log entries since after 5/15. No error was found either. We had enabled debug on the scheduler, restarted splunkd, and problem persists. Scheduler.log still has no new entries since after 5/15. At this point, I am not sure where else to look to further trouble shooting this issue.
I have enabled DEBUG by manually editing $SPLUNK_HOME/etc/log.cfg, and flipped the following parameter from INFO to DEBUG:
Before:
category.SavedSplunker=INFO,scheduler
After:
category.SavedSplunker=DEBUG,scheduler
Restarted splunkd, but still no new entries in the scheduler.log.
... View more