I upgraded from Splunk Enterprise 5.0.5 to 6.0 and this error is showing in logs.
Is it anything to worry about?
file : splunkd.log
10-04-2013 12:40:29.441 +0100 ERROR DispatchThread - Error reading runtime settings: File does not exist
This is a benign error - per the developers this should be suppressed and not displayed.
This should be fixed in an upcoming version of 6.x.
Brian
In-process searches are somewhat rare and usually not wanted. All valid cases for in-process searches I can think of involve the UI. (Usual cases are historical |metadata searches and post process actions in advanced xml dashboards). Therefore, although we suppressed this message within main splunkd (for splunkd.log) in 6.1.x, we should probably investigate why it is occurring in detail.
This message essentially means that there wasn't a file talking about autopause and autocancel times provided to a search. That's correct for scheduled searches so in 6.0.x we tried to change scheduled searches to never complain this way. If this message comes in splunkd.log, that means it's being produced by an in-process search.
Any news about that?
I just installed 6.1.0 and this error occurs in the splunkd.log every 10 min
Looking at python.log
, you'll see exactly what process is trying to spawn. Running that same process as the user splunk
should tell you if it works.
In my case, it seems that my splunkd
process was running as root
(oops!), but that the script that is spawned off runs as user splunk
.
The handoff takes place via a file in $SPLUNK/var/runs/plunk/dispatch
, which has the following permissions
drwx------ 410 splunk splunk 102400 Jan 30 12:23 dispatch
that's fine and should not lead to trouble. However, for each run, a new directory is created that is owned by root:root
, and which has restrictive permissions:
drwx------ 2 root root 4096 Jan 30 11:48 SummaryDirector_1391100527.76479
Just restarting splunkd
with a non-privileged user will break lots of stuff in terms of permissions in the filesystem, but I am currently testing that.
In the meanwhile, Splunk should probably consider updating code to reflect that the SummaryDirector directories are created with a nonprivileged user also. I will submit a bug report that includes this information.
This Fix Worked
Okay-- it took a bunch of effort, but I figured out that when Splunk calls a script, they include LD_LIBRARY_PATH
in the environment. My scripted action was calling another Python script, and the system's Python interpreter in /usr/bin/python
obeyed the variable, thus including /opt/splunk/lib
in the search path for the dynamic linker. The system-provided interpreter expected a different version of libssl
and libcrypto
than what Splunk ships with, which caused it not to execute.
Updating /opt/splunk/etc/apps/search/bin/runshellscript.py
to explicitly unset the LD_LIBRARY_PATH
environment variable just prior to firing off the script made the problem go away. Doing so probably voids some form of warrantee, but if you want to give it a go, edit that file and around line 192 (in between else:
and p = subprocess.Popen...
), add
del os.environ["LD_LIBRARY_PATH"]
Good luck.
Did anyone hear back from Splunk on this yet?
I'm getting the same err on Splunk 7.0
DispatchThread - Error reading runtime settings: File :/runtime.csv does not exist"
Any idea on how to fix it?
We are getting this error as well after updating to Splunk 7
Hi everyone will keep track on this post, our splunk instance also have this error
Also seeing the error
This is an old thread but seeing the same thing after upgrading to 6.0...
Nick
I suspect this is a bug and will need the Splunk devs to look at it - how can I file it as a bug request ?
Any luck on debugging this? We are hitting the same error on a search head after a 5-> 6 upgrade
can find what file is missing either. thousands of these errors
I thought enableing some enteries in the /opt/splunk/etc/log.cfg file would resolve this issue, but it did not...All enteries below were commented out before hand. Enabling of debug log did not yield any further information.
'# category.DispatchCommand no longer exists to get the same logging you can turn on these categories:
category.BundleReplicatorThread=INFO
category.DispatchCommandProcessor=INFO
category.DispatchCommandUtils=INFO
category.DispatchManager=INFO
category.DispatchProcess=INFO
category.DispatchSearchMetadata=INFO
category.DispatchThread=INFO
category.ISplunkDispatch=INFO
category.ProviderQueue=INFO
category.SearchStateListener=INFO
10-11-2013 17:42:15.442 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
10-11-2013 17:42:15.442 INFO DispatchThread - required fields list = *
10-11-2013 17:42:15.442 INFO ProviderQueue - Stream search:
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between previews will be last preview duration * 3.000000
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between reduces will be last reduce duration * 3.000000
10-11-2013 17:42:15.442 ERROR DispatchThread - Error reading runtime settings: File does not exist
10-11-2013 17:42:15.442 DEBUG DispatchThread - Field summary update frequency = 0.000000, max period = 60
10-11-2013 17:42:15.442 DEBUG DispatchThread - Reduction frequency = 10
10-11-2013 17:42:15.442 INFO DispatchThread - Disk quota = 10485760000
I also am having issues with this log entry. Is this something I should ignore? Is there a way to suppress this error?
10-11-2013 15:56:17.796 +0000 ERROR DispatchThread - Error reading runtime settings: File does not exist