I had this problem recently and it was for a tricky/silly reason. I got tired of the
dispatch directory being tied to the root volume and getting
The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch errors, so i created a 10G volume and mounted it over dispatch BUT I neglected to make it writable by the user running splunkd (i.e. "splunk"). In such a situation, 14 searches will start, but not really, and none will be able to complete so you get hung. I discovered the problem by going to the search head CLI and doing this (because I could not search against _*):
tail -f $SPLUNK_HOME/var/log/splunk/*
Very quickly I saw logs like this:
10-21-2016 12:02:10.208 -0400 ERROR SearchScheduler - failed to rm -r /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD54740dfff07b17ef1_at_1477065699_0: No such file or directory
In other words, it was trying to remove files that it was not able to create. OOPS! A simple
chmod later and all was good again.
Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled summary searches, the apps like deployment-monitor etc ....
The maximum number of searches is proportional of the number of cores on the system, so you could improve the hardware.
Thanks Yannk, But i haven't created any searches like this. How to debug the issue? i have used appencols for many of my searches and when i count the searches using append or appendcols and there are 5.. is that the reason ?
About the job summary :
you have some funny searches with just a "|" in it.
please find the admin of the SessionApp and tell him to stop running those searches.
about the hardware :
i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores.
Sad, you can barely run 2 realtime searches with that.
FYI the base hardware is 2 quad cores on a physical server. http://docs.splunk.com/Documentation/Splunk/5.0.1/Installation/Referencehardware
Yann has already answered this. You just have searches running you haven't spotted. Also bear in mind that on startup a lot of scheduled searches tend to fire which can have an impact. Frankly if you're running two instances on a 2 core machine you should just accept that you're going to receive these messages.
i have installed 2 splunk instances in the same VM.. and it have 2 cpu cores. Eve if i am using one instance at a time, the very first search itself is giving this message and it is slowing down my searches..
is this a splunk bug, or do i have to look into any of my configs? i went through limits.conf as well i didn't find anything wrong there also..
my splunk app is under testing and i am using a Virtual machine for the same. i have 2 cpu cores .. i tried changing the default values in authorize.conf.
my issue is as soon as i start the splunk instance , i am facing this error in the first search itself.