topic Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8 in Splunk Search

The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Tue, 22 Jan 2013 05:32:17 GMT

HI ,
Even if i just started my splunk instance, my views are loading with this error. I am sure that only one search is currently running. How can i troubleshoot this?
Please help
Thank you

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

yannK — Wed, 23 Jan 2013 17:11:29 GMT

Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled summary searches, the apps like deployment-monitor etc ....

The maximum number of searches is proportional of the number of cores on the system, so you could improve the hardware.
see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Accommodatemanysimultaneoussearches

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Fri, 25 Jan 2013 05:12:12 GMT

my splunk app is under testing and i am using a Virtual machine for the same. i have 2 cpu cores .. i tried changing the default values in authorize.conf.
my issue is as soon as i start the splunk instance , i am facing this error in the first search itself.

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

bellaed — Fri, 25 Jan 2013 06:49:55 GMT

Is there any limit for searches in a view? Currently i have 2 pages with 6 panels each.If I include 12 panels in one page, will the searches become slow?

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

yannK — Fri, 25 Jan 2013 16:35:15 GMT

no the extra searches will simply be skipped.

see answer for the system wide search limit on the other question :
http://splunk-base.splunk.com/answers/73074/limit-for-searches-in-a-page

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Mon, 28 Jan 2013 11:46:07 GMT

i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores. Eve if i am using one instance at a time, the very first search itself is giving this message and it is slowing down my searches..
is this a splunk bug, or do i have to look into any of my configs? i went through limits.conf as well i didn't find anything wrong there also..
please help

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

Drainy — Mon, 28 Jan 2013 12:01:33 GMT

Yann has already answered this. You just have searches running you haven't spotted. Also bear in mind that on startup a lot of scheduled searches tend to fire which can have an impact. Frankly if you're running two instances on a 2 core machine you should just accept that you're going to receive these messages.

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Mon, 28 Jan 2013 12:35:54 GMT

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Mon, 28 Jan 2013 12:37:31 GMT

This is my output when i tried reading job inspector...
i aouldnt able to find the issue with this input.
Somebody pls help, whats wrong with this

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

Drainy — Mon, 28 Jan 2013 12:39:05 GMT

and what is the output if you expand it to include all apps and all owners?

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Mon, 28 Jan 2013 12:45:05 GMT

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

yannK — Mon, 28 Jan 2013 17:55:42 GMT

About the job summary :
you have some funny searches with just a "|" in it.
please find the admin of the SessionApp and tell him to stop running those searches.

about the hardware :

i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores.

Sad, you can barely run 2 realtime searches with that.

FYI the base hardware is 2 quad cores on a physical server. http://docs.splunk.com/Documentation/Splunk/5.0.1/Installation/Referencehardware

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj — Tue, 29 Jan 2013 04:16:22 GMT

Thanks Yannk, But i haven't created any searches like this. How to debug the issue? i have used appencols for many of my searches and when i count the searches using append or appendcols and there are 5.. is that the reason ?

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

vqd361 — Fri, 12 Dec 2014 22:32:22 GMT

@smolcj, the app named Infa-Session is the one generating those queries. Disable the app to stop the searches.

Re: The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

woodcock — Sat, 22 Oct 2016 14:02:14 GMT

I had this problem recently and it was for a tricky/silly reason. I got tired of the dispatch directory being tied to the root volume and getting The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch errors, so i created a 10G volume and mounted it over dispatch BUT I neglected to make it writable by the user running splunkd (i.e. "splunk"). In such a situation, 14 searches will start, but not really, and none will be able to complete so you get hung. I discovered the problem by going to the search head CLI and doing this (because I could not search against _*):

tail -f $SPLUNK_HOME/var/log/splunk/*

Very quickly I saw logs like this:

10-21-2016 12:02:10.208 -0400 ERROR SearchScheduler - failed to rm -r /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD54740dfff07b17ef1_at_1477065699_0: No such file or directory

In other words, it was trying to remove files that it was not able to create. OOPS! A simple chmod later and all was good again.