Relation between mongod and scheduled searches

[maurelio79]

Hi to all, is there some relation with mongod and scheduled searches?
In our environment we always had mongod disabled, recently we enable it and after enabling it we found out that some scheduled searches started to be scheduled (for example) every seconds instead every 5 minutes (as cron schedule).
This not happens always but specially in some range time. Below you can see a table with columns _time and scheduled time for a particular search:

_time                      scheduled_time

2019-02-22 11:23:59.224 02/22/2019 11:20:00
2019-02-22 11:23:58.217 02/22/2019 11:20:00
2019-02-22 11:23:57.212 02/22/2019 11:20:00
2019-02-22 11:23:56.206 02/22/2019 11:20:00
2019-02-22 11:23:55.201 02/22/2019 11:20:00
2019-02-22 11:23:54.195 02/22/2019 11:20:00
2019-02-22 11:23:53.188 02/22/2019 11:20:00
2019-02-22 11:23:52.184 02/22/2019 11:20:00
2019-02-22 11:23:51.177 02/22/2019 11:20:00
2019-02-22 11:23:50.171 02/22/2019 11:20:00
2019-02-22 11:23:49.165 02/22/2019 11:20:00
2019-02-22 11:23:48.159 02/22/2019 11:20:00
2019-02-22 11:23:47.155 02/22/2019 11:20:00
2019-02-22 11:23:46.149 02/22/2019 11:20:00
2019-02-22 11:23:45.143 02/22/2019 11:20:00
2019-02-22 11:23:44.136 02/22/2019 11:20:00
2019-02-22 11:23:43.130 02/22/2019 11:20:00
2019-02-22 11:23:42.124 02/22/2019 11:20:00
2019-02-22 11:23:41.116 02/22/2019 11:20:00
2019-02-22 11:23:40.109 02/22/2019 11:20:00
2019-02-22 11:23:39.104 02/22/2019 11:20:00

As you can see every second the search was scheduled.
The search is not made by us but by third part: what kind of information you need about the search in order to help me understand this issue?

Thanks and regards.

[lakshman239]

Did you look at the cron/scheduled time of the search? how long does the search take to complete? Are you seeing skipped searches or concurrent search limit hits?

[maurelio79]

Hi, thanks very much for reply. The cron schedule could be every 5 minutes or maybe every 3 minutes.
For example a search today run correctly at 11.38, with status "success", then it run again at 11.48 (even it's scheduled every 3 minutes) and it started to run with status "continued" about 50 times in 1 minute, logging also the message:

WARN SavedSplunker - Max alive instance_count=1 reached for savedsearch_id="nobody;saved_search_name"

Also now, for example, the same search at 12.39 (in Italy) got a status "continued" with scheduled time 12.15 and message reason="The maximum number of concurrent running jobs for this historical scheduled search on this instance has been reached"

[lakshman239]

You are hitting the limit of concurrent searches. Look at all scheduled searches and see if you can adjust them and also look at optimising long running searches. Also, do you really need to run every 3 or 5mins? [ because if the search takes more time to complete, you would hit concurrent limits]