HI ,
Even if i just started my splunk instance, my views are loading with this error. I am sure that only one search is currently running. How can i troubleshoot this?
Please help
Thank you
I had this problem recently and it was for a tricky/silly reason. I got tired of the dispatch
directory being tied to the root volume and getting The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch
errors, so i created a 10G volume and mounted it over dispatch BUT I neglected to make it writable by the user running splunkd (i.e. "splunk"). In such a situation, 14 searches will start, but not really, and none will be able to complete so you get hung. I discovered the problem by going to the search head CLI and doing this (because I could not search against _*):
tail -f $SPLUNK_HOME/var/log/splunk/*
Very quickly I saw logs like this:
10-21-2016 12:02:10.208 -0400 ERROR SearchScheduler - failed to rm -r /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD54740dfff07b17ef1_at_1477065699_0: No such file or directory
In other words, it was trying to remove files that it was not able to create. OOPS! A simple chmod
later and all was good again.
Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled summary searches, the apps like deployment-monitor etc ....
The maximum number of searches is proportional of the number of cores on the system, so you could improve the hardware.
see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Accommodatemanysimultaneoussearches
Thanks Yannk, But i haven't created any searches like this. How to debug the issue? i have used appencols for many of my searches and when i count the searches using append or appendcols and there are 5.. is that the reason ?
@smolcj, the app named Infa-Session is the one generating those queries. Disable the app to stop the searches.
About the job summary :
you have some funny searches with just a "|" in it.
please find the admin of the SessionApp and tell him to stop running those searches.
about the hardware :
i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores.
Sad, you can barely run 2 realtime searches with that.
FYI the base hardware is 2 quad cores on a physical server. http://docs.splunk.com/Documentation/Splunk/5.0.1/Installation/Referencehardware
and what is the output if you expand it to include all apps and all owners?
This is my output when i tried reading job inspector...
i aouldnt able to find the issue with this input.
Somebody pls help, whats wrong with this
Yann has already answered this. You just have searches running you haven't spotted. Also bear in mind that on startup a lot of scheduled searches tend to fire which can have an impact. Frankly if you're running two instances on a 2 core machine you should just accept that you're going to receive these messages.
i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores. Eve if i am using one instance at a time, the very first search itself is giving this message and it is slowing down my searches..
is this a splunk bug, or do i have to look into any of my configs? i went through limits.conf as well i didn't find anything wrong there also..
please help
no the extra searches will simply be skipped.
see answer for the system wide search limit on the other question :
http://splunk-base.splunk.com/answers/73074/limit-for-searches-in-a-page
Is there any limit for searches in a view? Currently i have 2 pages with 6 panels each.If I include 12 panels in one page, will the searches become slow?
my splunk app is under testing and i am using a Virtual machine for the same. i have 2 cpu cores .. i tried changing the default values in authorize.conf.
my issue is as soon as i start the splunk instance , i am facing this error in the first search itself.