Splunk Search

The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8

smolcj
Builder

HI ,
Even if i just started my splunk instance, my views are loading with this error. I am sure that only one search is currently running. How can i troubleshoot this?
Please help
Thank you

Tags (2)

woodcock
Esteemed Legend

I had this problem recently and it was for a tricky/silly reason. I got tired of the dispatch directory being tied to the root volume and getting The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch errors, so i created a 10G volume and mounted it over dispatch BUT I neglected to make it writable by the user running splunkd (i.e. "splunk"). In such a situation, 14 searches will start, but not really, and none will be able to complete so you get hung. I discovered the problem by going to the search head CLI and doing this (because I could not search against _*):

tail -f $SPLUNK_HOME/var/log/splunk/*

Very quickly I saw logs like this:

10-21-2016 12:02:10.208 -0400 ERROR SearchScheduler - failed to rm -r /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD54740dfff07b17ef1_at_1477065699_0: No such file or directory

In other words, it was trying to remove files that it was not able to create. OOPS! A simple chmod later and all was good again.

0 Karma

yannK
Splunk Employee
Splunk Employee

Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled summary searches, the apps like deployment-monitor etc ....

The maximum number of searches is proportional of the number of cores on the system, so you could improve the hardware.
see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Accommodatemanysimultaneoussearches

smolcj
Builder

Thanks Yannk, But i haven't created any searches like this. How to debug the issue? i have used appencols for many of my searches and when i count the searches using append or appendcols and there are 5.. is that the reason ?

0 Karma

vqd361
Path Finder

@smolcj, the app named Infa-Session is the one generating those queries. Disable the app to stop the searches.

0 Karma

yannK
Splunk Employee
Splunk Employee

About the job summary :
you have some funny searches with just a "|" in it.
please find the admin of the SessionApp and tell him to stop running those searches.

about the hardware :

i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores.

Sad, you can barely run 2 realtime searches with that.

FYI the base hardware is 2 quad cores on a physical server. http://docs.splunk.com/Documentation/Splunk/5.0.1/Installation/Referencehardware

smolcj
Builder

alt text

0 Karma

Drainy
Champion

and what is the output if you expand it to include all apps and all owners?

0 Karma

smolcj
Builder

This is my output when i tried reading job inspector...
i aouldnt able to find the issue with this input.
Somebody pls help, whats wrong with this

0 Karma

smolcj
Builder

alt text

0 Karma

Drainy
Champion

Yann has already answered this. You just have searches running you haven't spotted. Also bear in mind that on startup a lot of scheduled searches tend to fire which can have an impact. Frankly if you're running two instances on a 2 core machine you should just accept that you're going to receive these messages.

0 Karma

smolcj
Builder

i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores. Eve if i am using one instance at a time, the very first search itself is giving this message and it is slowing down my searches..
is this a splunk bug, or do i have to look into any of my configs? i went through limits.conf as well i didn't find anything wrong there also..
please help

0 Karma

yannK
Splunk Employee
Splunk Employee

no the extra searches will simply be skipped.

see answer for the system wide search limit on the other question :
http://splunk-base.splunk.com/answers/73074/limit-for-searches-in-a-page

0 Karma

bellaed
Path Finder

Is there any limit for searches in a view? Currently i have 2 pages with 6 panels each.If I include 12 panels in one page, will the searches become slow?

0 Karma

smolcj
Builder

my splunk app is under testing and i am using a Virtual machine for the same. i have 2 cpu cores .. i tried changing the default values in authorize.conf.
my issue is as soon as i start the splunk instance , i am facing this error in the first search itself.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...