Why are our Splunk Forwarders each logging "ERROR DiskMon - None such on disk: /opt/splunkforwarder/var/run/splunk/dispatch" 144 times a day?

[starks951]

We are seeing these errors in the forwarders splunkd.log from every Splunk forwarder we have 144 times per 24hr period (every ten mins) per server. All servers are running Splunk Light Forwarder 6.2.1, 6.2.2, and 6.2.3 across the enterprise.

on Linux hosts it looks like this

07-28-2015 19:48:35.831 -0500 ERROR DiskMon - None such on disk: /opt/splunkforwarder/var/run/splunk/dispatch

and on windows like this

07-28-2015 19:48:34.228 -0500 ERROR DiskMon - None such on disk: F:\Program Files\SplunkUniversalForwarder\var\run\splunk\dispatch

The only thing I can think of is that something is trying to run splunkd clean-dispatch and the system can't find the dir, but I can't find anything in our deployment server that would be configured to run this. I have seen these errors in a few other logs posted here, but the issues seem to be about a server trying to bind to a port already in use on Win7 (not our issue here).

Anyone have ANY idea why this might be happening?

[Ellen · Accepted solution]

What you are seeing in 6.2 - 6.2.6 is a known issue, SPL-109387 where both UF and LWF will log this benign error every 10 minutes.

ERROR DiskMon - None such on disk: .../splunkforwarder/var/run/splunk/dispatch

This is related when the app .../splunkforwarder/etc/apps/introspection_generator_addon has been enabled and information relating to disk object partitions is attempted to be retrieved.

Below are 3 different workarounds:

Option 1:
Edit ../splunkforwarder/etc/system/local/server.conf and add the following entry:

[introspection:generator:disk_objects__partitions]
disabled = true

Option 2:
In ../splunkforwarder/etc/log.cfg (requires restart),
Increase the logging level of category.DiskMon=INFO to

category.DiskMon=CRIT

Option 3:
3) create on that UF/LWF an empty directory called... /splunkforwarder/var/run/splunk/dispatch

[USPSSplunkSuppo]

I suspect the introspection app cannot differentiate between forwarder and other Splunk device types such as Indexers, Search Heads, etc.

My workaround was to: mkdir $SPLUNK_HOME/var/run/splunk/dispatch

But then I only currently have a small number of forwarders.