I upgraded from Splunk Enterprise 5.0.5 to 6.0 and this error is showing in logs.
Is it anything to worry about?
file : splunkd.log 10-04-2013 12:40:29.441 +0100 ERROR DispatchThread - Error reading runtime settings: File does not exist
In-process searches are somewhat rare and usually not wanted. All valid cases for in-process searches I can think of involve the UI. (Usual cases are historical |metadata searches and post process actions in advanced xml dashboards). Therefore, although we suppressed this message within main splunkd (for splunkd.log) in 6.1.x, we should probably investigate why it is occurring in detail.
This message essentially means that there wasn't a file talking about autopause and autocancel times provided to a search. That's correct for scheduled searches so in 6.0.x we tried to change scheduled searches to never complain this way. If this message comes in splunkd.log, that means it's being produced by an in-process search.
python.log, you'll see exactly what process is trying to spawn. Running that same process as the user
splunk should tell you if it works.
In my case, it seems that my
splunkd process was running as
root (oops!), but that the script that is spawned off runs as user
The handoff takes place via a file in
$SPLUNK/var/runs/plunk/dispatch, which has the following permissions
drwx------ 410 splunk splunk 102400 Jan 30 12:23 dispatch
that's fine and should not lead to trouble. However, for each run, a new directory is created that is owned by
root:root, and which has restrictive permissions:
drwx------ 2 root root 4096 Jan 30 11:48 SummaryDirector_1391100527.76479
splunkd with a non-privileged user will break lots of stuff in terms of permissions in the filesystem, but I am currently testing that.
In the meanwhile, Splunk should probably consider updating code to reflect that the SummaryDirector directories are created with a nonprivileged user also. I will submit a bug report that includes this information.
Okay-- it took a bunch of effort, but I figured out that when Splunk calls a script, they include
LD_LIBRARY_PATH in the environment. My scripted action was calling another Python script, and the system's Python interpreter in
/usr/bin/python obeyed the variable, thus including
/opt/splunk/lib in the search path for the dynamic linker. The system-provided interpreter expected a different version of
libcrypto than what Splunk ships with, which caused it not to execute.
/opt/splunk/etc/apps/search/bin/runshellscript.py to explicitly unset the
LD_LIBRARY_PATH environment variable just prior to firing off the script made the problem go away. Doing so probably voids some form of warrantee, but if you want to give it a go, edit that file and around line 192 (in between
p = subprocess.Popen...), add
I thought enableing some enteries in the /opt/splunk/etc/log.cfg file would resolve this issue, but it did not...All enteries below were commented out before hand. Enabling of debug log did not yield any further information.
'# category.DispatchCommand no longer exists to get the same logging you can turn on these categories:
10-11-2013 17:42:15.442 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
10-11-2013 17:42:15.442 INFO DispatchThread - required fields list = *
10-11-2013 17:42:15.442 INFO ProviderQueue - Stream search:
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between previews will be last preview duration * 3.000000
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between reduces will be last reduce duration * 3.000000
10-11-2013 17:42:15.442 ERROR DispatchThread - Error reading runtime settings: File does not exist
10-11-2013 17:42:15.442 DEBUG DispatchThread - Field summary update frequency = 0.000000, max period = 60
10-11-2013 17:42:15.442 DEBUG DispatchThread - Reduction frequency = 10
10-11-2013 17:42:15.442 INFO DispatchThread - Disk quota = 10485760000
I also am having issues with this log entry. Is this something I should ignore? Is there a way to suppress this error?
10-11-2013 15:56:17.796 +0000 ERROR DispatchThread - Error reading runtime settings: File does not exist