Getting Data In

Why are our Splunk Forwarders each logging "ERROR DiskMon - None such on disk: /opt/splunkforwarder/var/run/splunk/dispatch" 144 times a day?

starks951
Explorer

We are seeing these errors in the forwarders splunkd.log from every Splunk forwarder we have 144 times per 24hr period (every ten mins) per server. All servers are running Splunk Light Forwarder 6.2.1, 6.2.2, and 6.2.3 across the enterprise.

on Linux hosts it looks like this

07-28-2015 19:48:35.831 -0500 ERROR DiskMon - None such on disk: /opt/splunkforwarder/var/run/splunk/dispatch

and on windows like this

07-28-2015 19:48:34.228 -0500 ERROR DiskMon - None such on disk: F:\Program Files\SplunkUniversalForwarder\var\run\splunk\dispatch

The only thing I can think of is that something is trying to run splunkd clean-dispatch and the system can't find the dir, but I can't find anything in our deployment server that would be configured to run this. I have seen these errors in a few other logs posted here, but the issues seem to be about a server trying to bind to a port already in use on Win7 (not our issue here).

Anyone have ANY idea why this might be happening?

Tags (2)
0 Karma
1 Solution

Ellen
Splunk Employee
Splunk Employee

What you are seeing in 6.2 - 6.2.6 is a known issue, SPL-109387 where both UF and LWF will log this benign error every 10 minutes.

ERROR DiskMon - None such on disk: .../splunkforwarder/var/run/splunk/dispatch

This is related when the app .../splunkforwarder/etc/apps/introspection_generator_addon has been enabled and information relating to disk object partitions is attempted to be retrieved.

Below are 3 different workarounds:

Option 1:
Edit ../splunkforwarder/etc/system/local/server.conf and add the following entry:

[introspection:generator:disk_objects__partitions]
disabled = true

Option 2:
In ../splunkforwarder/etc/log.cfg (requires restart),
Increase the logging level of category.DiskMon=INFO to

category.DiskMon=CRIT

Option 3:
3) create on that UF/LWF an empty directory called... /splunkforwarder/var/run/splunk/dispatch

View solution in original post

Ellen
Splunk Employee
Splunk Employee

What you are seeing in 6.2 - 6.2.6 is a known issue, SPL-109387 where both UF and LWF will log this benign error every 10 minutes.

ERROR DiskMon - None such on disk: .../splunkforwarder/var/run/splunk/dispatch

This is related when the app .../splunkforwarder/etc/apps/introspection_generator_addon has been enabled and information relating to disk object partitions is attempted to be retrieved.

Below are 3 different workarounds:

Option 1:
Edit ../splunkforwarder/etc/system/local/server.conf and add the following entry:

[introspection:generator:disk_objects__partitions]
disabled = true

Option 2:
In ../splunkforwarder/etc/log.cfg (requires restart),
Increase the logging level of category.DiskMon=INFO to

category.DiskMon=CRIT

Option 3:
3) create on that UF/LWF an empty directory called... /splunkforwarder/var/run/splunk/dispatch

USPSSplunkSuppo
Explorer

I suspect the introspection app cannot differentiate between forwarder and other Splunk device types such as Indexers, Search Heads, etc.

My workaround was to: mkdir $SPLUNK_HOME/var/run/splunk/dispatch

But then I only currently have a small number of forwarders.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...