We are seeing these errors in the forwarders splunkd.log from every Splunk forwarder we have 144 times per 24hr period (every ten mins) per server. All servers are running Splunk Light Forwarder 6.2.1, 6.2.2, and 6.2.3 across the enterprise.
on Linux hosts it looks like this
07-28-2015 19:48:35.831 -0500 ERROR DiskMon - None such on disk: /opt/splunkforwarder/var/run/splunk/dispatch
and on windows like this
07-28-2015 19:48:34.228 -0500 ERROR DiskMon - None such on disk: F:\Program Files\SplunkUniversalForwarder\var\run\splunk\dispatch
The only thing I can think of is that something is trying to run splunkd clean-dispatch
and the system can't find the dir, but I can't find anything in our deployment server that would be configured to run this. I have seen these errors in a few other logs posted here, but the issues seem to be about a server trying to bind to a port already in use on Win7 (not our issue here).
Anyone have ANY idea why this might be happening?
What you are seeing in 6.2 - 6.2.6 is a known issue, SPL-109387 where both UF and LWF will log this benign error every 10 minutes.
ERROR DiskMon - None such on disk: .../splunkforwarder/var/run/splunk/dispatch
This is related when the app .../splunkforwarder/etc/apps/introspection_generator_addon has been enabled and information relating to disk object partitions is attempted to be retrieved.
Below are 3 different workarounds:
Option 1:
Edit ../splunkforwarder/etc/system/local/server.conf and add the following entry:
[introspection:generator:disk_objects__partitions]
disabled = true
Option 2:
In ../splunkforwarder/etc/log.cfg (requires restart),
Increase the logging level of category.DiskMon=INFO to
category.DiskMon=CRIT
Option 3:
3) create on that UF/LWF an empty directory called... /splunkforwarder/var/run/splunk/dispatch
What you are seeing in 6.2 - 6.2.6 is a known issue, SPL-109387 where both UF and LWF will log this benign error every 10 minutes.
ERROR DiskMon - None such on disk: .../splunkforwarder/var/run/splunk/dispatch
This is related when the app .../splunkforwarder/etc/apps/introspection_generator_addon has been enabled and information relating to disk object partitions is attempted to be retrieved.
Below are 3 different workarounds:
Option 1:
Edit ../splunkforwarder/etc/system/local/server.conf and add the following entry:
[introspection:generator:disk_objects__partitions]
disabled = true
Option 2:
In ../splunkforwarder/etc/log.cfg (requires restart),
Increase the logging level of category.DiskMon=INFO to
category.DiskMon=CRIT
Option 3:
3) create on that UF/LWF an empty directory called... /splunkforwarder/var/run/splunk/dispatch
I suspect the introspection app cannot differentiate between forwarder and other Splunk device types such as Indexers, Search Heads, etc.
My workaround was to: mkdir $SPLUNK_HOME/var/run/splunk/dispatch
But then I only currently have a small number of forwarders.