in splunkd.log a lot of warnings : DispatchCommand - could not read metadata file

[imrago]

In my splunkd.log (v4.1) I have a lot of warnings like these :

04-13-2010 00:05:19.676 WARN  DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/1271116501.1/metadata.csv
04-13-2010 00:05:19.677 WARN  DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/1271116742.1/metadata.csv
04-13-2010 00:13:50.395 WARN  DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/1271117581.1/metadata.csv
04-13-2010 00:13:50.395 WARN  DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/1271117162.1/metadata.csv

What could be the reason for these warnings?

[fleXible]

I found a solution to my specific breed of the problem. After toying around with the Splunk_SA_CIM and SplunkAppForWebAnalytics, which both define the Web datamodel, my log quickly filled with these messages:

08-14-2016 09:40:58.919 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:40:58.919 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv
08-14-2016 09:41:28.922 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:41:28.923 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv
08-14-2016 09:41:58.918 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD5cd22bc27c7bb1b18_at_1471146000_98/metadata.csv
08-14-2016 09:41:58.918 +0200 WARN  DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/metadata.csv

The reason for this error message in my special case was, by often restarting the splunk service, it was unable to finish correctly writing out the dispatch data and just left corrupt (empty) files there instead.

ll scheduler__nobody__SplunkAppForWebAnalytics__RMD56b5a72de0a2a981e_at_1471146000_97/
total 36K
prw------- 1 root root    0 Aug 14 05:40 alive.token|
-rw------- 1 root root    0 Aug 14 05:40 args.txt
-rw------- 1 root root    0 Aug 14 05:53 audited
-rw------- 1 root root  28K Aug 14 05:40 info.csv
-rw------- 1 root root    0 Aug 14 05:40 metadata.csv
-rw------- 1 root root    0 Aug 14 05:40 peers.csv
-rw------- 1 root root    0 Aug 14 05:40 pipeline_sets
-rw------- 1 root root    0 Aug 14 05:40 request.csv
-rw------- 1 root root    0 Aug 14 05:40 search.log
-rw------- 1 root root 6.8K Aug 14 05:40 status.csv

After removing the corrupt directories, the error messages went away with them.

[Mick]

This is one explanation, another is that there is a known bug in 4.1.2 & 4.1.3 where Splunk tries to access a results file before it is actually created. It's not really anything to be concerned about, unless you're actually noticing a problem with loading search results or accessing saved results objects.

It will be resolved in an upcoming release

[imrago · Accepted solution]

Finally I have found the underlaying problem. From the cron the searches in splunk where executed as root user, and the owner of those files in /opt/splunk/var/run/splunk/dispatch/.... was root, which in turn caused the error messages in previous post.

Changed in cron root to splunk and the errors disappeared.