Join the Conversation
- Find Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Good setting for max_searches_per_cpu?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I'm getting this warning every hour, on top of the hour, when apparently quite a few scheduled searches are triggered.
The system is approaching the maximum number of historical searches that can be run concurrently. current=19 maximum=20
Now, I understand that this does not necessarily cause a problem, since the searches will be slightly postponed. Correct? However, the CPU usage on all cores for both search heads is very close to 0%, constantly. So it seems that increasing max_searches_per_cpu slightly could be appropriate, but I can't find documentation explaining what good values are.
The CPU model is Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz.
There are 4 cores per search head and 8GB RAM.
Any opinions?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The default setting is max_searches_per_cpu=4 and is already optimized. Increasing it may cause your searches to slow down or worse.
The best solution to avoid to reach the max, is to change the schedule of your searches. Spread them other different intervals.
Example, for an hourly search over the last hour :
cron = 0 * * * * earliest=-1h latest=now
can be replaced by a search that starts 5min after the hour :
cron=5 * * * * earliest=-1h@h latest=@h
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The default setting is max_searches_per_cpu=4 and is already optimized. Increasing it may cause your searches to slow down or worse.
The best solution to avoid to reach the max, is to change the schedule of your searches. Spread them other different intervals.
Example, for an hourly search over the last hour :
cron = 0 * * * * earliest=-1h latest=now
can be replaced by a search that starts 5min after the hour :
cron=5 * * * * earliest=-1h@h latest=@h
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
On the other hand, I do feel that the cpus are heavily under-utilised. Whenever I get this warning, the cpus are nearly idle.
So, unless the problem is between the search head and the indexers, I don't really see why a slight increase would be a problem... Doubling the cores would remove this warning, but would also result in more idle cpus.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
to have an idea of the scheduling searches, take a look at the SOS app
http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk
the searches dashboard shows the scheduling and skipped searches.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok, thanks!
Rescheduling sounds doable. The only hitch is that users can schedule searches, so they may not be aware of this issue.
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Announcing Modern Navigation: A New Era of Splunk User Experience
Modernize your Splunk Apps – Introducing Python 3.13 in Splunk
Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...
