Splunk Search

Relation between mongod and scheduled searches

maurelio79
Communicator

Hi to all, is there some relation with mongod and scheduled searches?
In our environment we always had mongod disabled, recently we enable it and after enabling it we found out that some scheduled searches started to be scheduled (for example) every seconds instead every 5 minutes (as cron schedule).
This not happens always but specially in some range time. Below you can see a table with columns _time and scheduled time for a particular search:

_time                      scheduled_time

2019-02-22 11:23:59.224 02/22/2019 11:20:00
2019-02-22 11:23:58.217 02/22/2019 11:20:00
2019-02-22 11:23:57.212 02/22/2019 11:20:00
2019-02-22 11:23:56.206 02/22/2019 11:20:00
2019-02-22 11:23:55.201 02/22/2019 11:20:00
2019-02-22 11:23:54.195 02/22/2019 11:20:00
2019-02-22 11:23:53.188 02/22/2019 11:20:00
2019-02-22 11:23:52.184 02/22/2019 11:20:00
2019-02-22 11:23:51.177 02/22/2019 11:20:00
2019-02-22 11:23:50.171 02/22/2019 11:20:00
2019-02-22 11:23:49.165 02/22/2019 11:20:00
2019-02-22 11:23:48.159 02/22/2019 11:20:00
2019-02-22 11:23:47.155 02/22/2019 11:20:00
2019-02-22 11:23:46.149 02/22/2019 11:20:00
2019-02-22 11:23:45.143 02/22/2019 11:20:00
2019-02-22 11:23:44.136 02/22/2019 11:20:00
2019-02-22 11:23:43.130 02/22/2019 11:20:00
2019-02-22 11:23:42.124 02/22/2019 11:20:00
2019-02-22 11:23:41.116 02/22/2019 11:20:00
2019-02-22 11:23:40.109 02/22/2019 11:20:00
2019-02-22 11:23:39.104 02/22/2019 11:20:00

As you can see every second the search was scheduled.
The search is not made by us but by third part: what kind of information you need about the search in order to help me understand this issue?

Thanks and regards.

0 Karma

lakshman239
Influencer

Did you look at the cron/scheduled time of the search? how long does the search take to complete? Are you seeing skipped searches or concurrent search limit hits?

0 Karma

maurelio79
Communicator

Hi, thanks very much for reply. The cron schedule could be every 5 minutes or maybe every 3 minutes.
For example a search today run correctly at 11.38, with status "success", then it run again at 11.48 (even it's scheduled every 3 minutes) and it started to run with status "continued" about 50 times in 1 minute, logging also the message:

WARN SavedSplunker - Max alive instance_count=1 reached for savedsearch_id="nobody;saved_search_name"

Also now, for example, the same search at 12.39 (in Italy) got a status "continued" with scheduled time 12.15 and message reason="The maximum number of concurrent running jobs for this historical scheduled search on this instance has been reached"

0 Karma

lakshman239
Influencer

You are hitting the limit of concurrent searches. Look at all scheduled searches and see if you can adjust them and also look at optimising long running searches. Also, do you really need to run every 3 or 5mins? [ because if the search takes more time to complete, you would hit concurrent limits]

0 Karma
Get Updates on the Splunk Community!

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...