Installation

Error when upgrading from Splunk Enterprise 5.0.5 to 6.0.

Explorer

I upgraded from Splunk Enterprise 5.0.5 to 6.0 and this error is showing in logs.
Is it anything to worry about?

file : splunkd.log
10-04-2013 12:40:29.441 +0100 ERROR DispatchThread - Error reading runtime settings: File does not exist

 

Labels (2)

Splunk Employee
Splunk Employee

This is a benign error - per the developers this should be suppressed and not displayed.

This should be fixed in an upcoming version of 6.x.

Brian

Splunk Employee
Splunk Employee

In-process searches are somewhat rare and usually not wanted. All valid cases for in-process searches I can think of involve the UI. (Usual cases are historical |metadata searches and post process actions in advanced xml dashboards). Therefore, although we suppressed this message within main splunkd (for splunkd.log) in 6.1.x, we should probably investigate why it is occurring in detail.

0 Karma

Splunk Employee
Splunk Employee

This message essentially means that there wasn't a file talking about autopause and autocancel times provided to a search. That's correct for scheduled searches so in 6.0.x we tried to change scheduled searches to never complain this way. If this message comes in splunkd.log, that means it's being produced by an in-process search.

0 Karma

Explorer

Any news about that?
I just installed 6.1.0 and this error occurs in the splunkd.log every 10 min

0 Karma

Path Finder

Looking at python.log, you'll see exactly what process is trying to spawn. Running that same process as the user splunk should tell you if it works.

In my case, it seems that my splunkd process was running as root (oops!), but that the script that is spawned off runs as user splunk.

The handoff takes place via a file in $SPLUNK/var/runs/plunk/dispatch, which has the following permissions

drwx------ 410 splunk splunk 102400 Jan 30 12:23 dispatch

that's fine and should not lead to trouble. However, for each run, a new directory is created that is owned by root:root, and which has restrictive permissions:

drwx------   2 root   root     4096 Jan 30 11:48 SummaryDirector_1391100527.76479

Just restarting splunkd with a non-privileged user will break lots of stuff in terms of permissions in the filesystem, but I am currently testing that.

In the meanwhile, Splunk should probably consider updating code to reflect that the SummaryDirector directories are created with a nonprivileged user also. I will submit a bug report that includes this information.

Path Finder

This Fix Worked

0 Karma

Path Finder

Okay-- it took a bunch of effort, but I figured out that when Splunk calls a script, they include LD_LIBRARY_PATH in the environment. My scripted action was calling another Python script, and the system's Python interpreter in /usr/bin/python obeyed the variable, thus including /opt/splunk/lib in the search path for the dynamic linker. The system-provided interpreter expected a different version of libssl and libcrypto than what Splunk ships with, which caused it not to execute.

Updating /opt/splunk/etc/apps/search/bin/runshellscript.py to explicitly unset the LD_LIBRARY_PATH environment variable just prior to firing off the script made the problem go away. Doing so probably voids some form of warrantee, but if you want to give it a go, edit that file and around line 192 (in between else: and p = subprocess.Popen...), add

del os.environ["LD_LIBRARY_PATH"]

Good luck.

Explorer

Did anyone hear back from Splunk on this yet?

0 Karma

Explorer

I'm getting the same err on Splunk 7.0

DispatchThread - Error reading runtime settings: File :/runtime.csv does not exist"

Any idea on how to fix it?

0 Karma

We are getting this error as well after updating to Splunk 7

0 Karma

Communicator

Hi everyone will keep track on this post, our splunk instance also have this error

0 Karma

Contributor

Also seeing the error

0 Karma

Path Finder

This is an old thread but seeing the same thing after upgrading to 6.0...

Nick

0 Karma

Explorer

I suspect this is a bug and will need the Splunk devs to look at it - how can I file it as a bug request ?

0 Karma

Path Finder

Any luck on debugging this? We are hitting the same error on a search head after a 5-> 6 upgrade

Communicator

can find what file is missing either. thousands of these errors

0 Karma

Engager

I thought enableing some enteries in the /opt/splunk/etc/log.cfg file would resolve this issue, but it did not...All enteries below were commented out before hand. Enabling of debug log did not yield any further information.

'# category.DispatchCommand no longer exists to get the same logging you can turn on these categories:
category.BundleReplicatorThread=INFO

category.DispatchCommandProcessor=INFO

category.DispatchCommandUtils=INFO

category.DispatchManager=INFO

category.DispatchProcess=INFO

category.DispatchSearchMetadata=INFO

category.DispatchThread=INFO

category.ISplunkDispatch=INFO

category.ProviderQueue=INFO

category.SearchStateListener=INFO


10-11-2013 17:42:15.442 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
10-11-2013 17:42:15.442 INFO DispatchThread - required fields list = *
10-11-2013 17:42:15.442 INFO ProviderQueue - Stream search:
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between previews will be last preview duration * 3.000000
10-11-2013 17:42:15.442 DEBUG DispatchThread - To enforce duty cycle, minimum time between reduces will be last reduce duration * 3.000000
10-11-2013 17:42:15.442 ERROR DispatchThread - Error reading runtime settings: File does not exist
10-11-2013 17:42:15.442 DEBUG DispatchThread - Field summary update frequency = 0.000000, max period = 60
10-11-2013 17:42:15.442 DEBUG DispatchThread - Reduction frequency = 10
10-11-2013 17:42:15.442 INFO DispatchThread - Disk quota = 10485760000

0 Karma

Engager

I also am having issues with this log entry. Is this something I should ignore? Is there a way to suppress this error?

10-11-2013 15:56:17.796 +0000 ERROR DispatchThread - Error reading runtime settings: File does not exist