Why am I getting a HTTP 503 error when using threads to call oneShotSearch and maximum number of concurrent historical searches?

[Aweave15]

Hi All,

I am using the Java splunk api service to make oneShotSearch calls for service data.
HTTP 503 response: Search not executed: The maximum number of concurrent historical searches on this instance has been reached.", concurrency_category="historical", concurrency_context="instance-wide", current_concurrency=34, concurrency_limit=34

What I am doing is creating multiple threads which perform the Service#oneshotSearch in given time intervals. It runs normally, but eventually, I get the HTTP 503 above error. I am not that familiar with Splunk (mainly focusing on the Java code) so could someone please elaborate on what this exactly means?

I am confused what exactly instance is referring to, is this referring to a specific host included in the queries, or is this referring to my own user searches once I am logged in? I do not have 34 threads, so are they getting backed up and queued through the api?

If anyone could please provide me with some details on this error to help troubleshoot, I would greatly appreciate it. Thank You!

[493669]

Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled searches running
there are 2 limits that may kick in :
- quota of number of concurrent searches per user (see the roles, and authorize.conf)
- quota of the system maximum number of concurrent searches (see number of cores and limits.conf)

And those limits are different for historical searches and real time searches.
see http://wiki.splunk.com/Community:TroubleshootingSearchQuotas
have a look at https://answers.splunk.com/answers/607068/the-maximum-number-of-concurrent-historical-search.html