How to resolve this Error in search: The maximum number of concurrent historical searches for this user has been reached

[dhimanv]

Hello,

I am getting error below when trying to search the data in Splunk SearchHead:

ERROR SearchScheduler - The maximum number of concurrent historical searches for this user based on their role quota has been reached

I tried to change the parameters as below

scheduler

max_searches_per_cpu from 25 to 50

search

max_searches_per_cpu from 6 to 10

But too seems not working and even generating high LOAD.

Could you please suggest how I can fix this issue? I am using Splunk Enterprise 8.0.5 version (I know its out of date  but still need help to figure out issue to plan upgrade)

[PickleRick]

You have max_searches_per_cpu at 25 or 50? It's supposed to be 1 by default. Sometimes you can set it to something more, but not by that factor! Are you sure you're not talking about a different parameter?

Anyway, as the message says - your user is exceeding quota enforced by his role, not the global configuration.

So if you have some "spare room" within your resources you can raise the user's limits.

[dhimanv]

@PickleRick  Thanks for your response.

Sorry it was typo, i changed the max_searches_perc from 25 to 50 and max_searches_per_cpu from 6 to 10 but still getting the error. I have 8 vCPU with and 32GB of instances in both Indexers and searchhead.

Could you advice which params I can tweak to fix the error and able to search the data.

Thanks once again. 

[PickleRick]

If you're raising max_searches_per_cpu to 10 (even 6 was a relatively high value), you have to expect trouble.

But as I said before - your "problem" is not in server parameters, but rather in limitations applied to your user which is running the searches (or on whose behalf the searches are run).

Check the user's roles and adjust the quota for max searches there. On user's level, not in global server configuration.

BTW, 8vCPU is a relatively small machine. The reference indexer has minimum of 12 cores and searchhead - 16.

[dhimanv]

Thank you.

I was looking for setting under SearchHead => Settings => Roles/Users but unable to locate any setting related to concurrent searches for which error is generated. Is this something which I need to update from CLI configuration of Splunk?

Sorry if I am asking something very common or stupid, I am trying to understand Splunk.

[PickleRick]

Settings -> Roles

Edit particular role

5th tab - Resources

There you have two limit categories - one is applied as a total to all users of a given role and another is applied to each user separately.

I suppose it's the latter that's the cause of your error here.