Splunk Search

Good setting for max_searches_per_cpu?

echalex
Builder

Hi,

I'm getting this warning every hour, on top of the hour, when apparently quite a few scheduled searches are triggered.

The system is approaching the maximum number of historical searches that can be run concurrently. current=19 maximum=20

Now, I understand that this does not necessarily cause a problem, since the searches will be slightly postponed. Correct? However, the CPU usage on all cores for both search heads is very close to 0%, constantly. So it seems that increasing max_searches_per_cpu slightly could be appropriate, but I can't find documentation explaining what good values are.

The CPU model is Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz.
There are 4 cores per search head and 8GB RAM.

Any opinions?

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

The default setting is max_searches_per_cpu=4 and is already optimized. Increasing it may cause your searches to slow down or worse.

The best solution to avoid to reach the max, is to change the schedule of your searches. Spread them other different intervals.

Example, for an hourly search over the last hour :
cron = 0 * * * * earliest=-1h latest=now

can be replaced by a search that starts 5min after the hour :
cron=5 * * * * earliest=-1h@h latest=@h

View solution in original post

dbylertbg
Path Finder

As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.

0 Karma

yannK
Splunk Employee
Splunk Employee

The default setting is max_searches_per_cpu=4 and is already optimized. Increasing it may cause your searches to slow down or worse.

The best solution to avoid to reach the max, is to change the schedule of your searches. Spread them other different intervals.

Example, for an hourly search over the last hour :
cron = 0 * * * * earliest=-1h latest=now

can be replaced by a search that starts 5min after the hour :
cron=5 * * * * earliest=-1h@h latest=@h

View solution in original post

dbylertbg
Path Finder

As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.

0 Karma

echalex
Builder

On the other hand, I do feel that the cpus are heavily under-utilised. Whenever I get this warning, the cpus are nearly idle.

So, unless the problem is between the search head and the indexers, I don't really see why a slight increase would be a problem... Doubling the cores would remove this warning, but would also result in more idle cpus.

0 Karma

yannK
Splunk Employee
Splunk Employee

to have an idea of the scheduling searches, take a look at the SOS app
http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk

the searches dashboard shows the scheduling and skipped searches.

0 Karma

echalex
Builder

Ok, thanks!

Rescheduling sounds doable. The only hitch is that users can schedule searches, so they may not be aware of this issue.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!