The error log had the answer. The new sysadmin had deleted the old sysadmin user. This killed all his saved searches. Which turned out to be nearly every search. I had to manually clone each search as myself. Tiring but it worked 😄
There is a global limit on the number of concurrently running searches set in limits.conf and based on the number of CPU cores you have, and there is a role-based quota on the number of running jobs a particular user can have, and includes scheduled jobs. This latter is much higher for an admin than for a power user.
I have somewhat given up.
I create a saved search.Then click schedule.
I choose a "Time range" of 2m@m "Start time" nothing entered in "Finish time".
And I set the "Cron schedule" to */2 * * * *.
My "Alert conditions" are "if the number of events" "is greater than" 0.
I tick "Include results in email".
And my "Trigger shell script" is a script that sends me an sms.
As soon as it runs, it switches scheduling off.
As in the manage search UI, I can see that the scheduled times now "None".
And if I click the saved search, the scheduling is no longer selected.
Well, thank you for the info, I am reading the documentation around limits.
I had 11 scheduled searches running, over the weekend.
Unfortunately, only 2 of them are running today.
Will spend some time reading documentation and let you know if I find anything.
Hi. Thanks for speady reply.
I opened all nine saved searches on separate tabs in browser.
Then scheduled all of them again this morning.
Splunk even remembers the scheduling settings, so I just had to click save.
And they all running fine.
Difference is I logged in as the admin user today.
Could have been a power user restriction, I am not sure.