Splunk Search

Search results in dispatch command message "The minimum free disk space (18446744073709551615MB) reached"

dccrain
New Member

Recently I migrated one of our indexers to a new machine.

Sometimes searches result in the below message despite there being plenty of space in the /opt/splunk mount and the minFreeSpace being set to 2000 in /opt/splunk/etc/system/local/server.conf.

Dispatch Command: The minimum free disk space (18446744073709551615MB) reached for /opt/splunk/var/run/splunk/dispatch dispite [diskUsage]minFreeSpace = 2000
0 Karma

dccrain
New Member

Found the problem: the /opt/splunk/var/run/splunk/dispatch dir was a symlink to a different storage array on the old machine, and was broken when rsync-ing from the the old host to the replacement, which has a single large /opt/ mount for splunk. Deleting the broken symlink and making a proper dir and then copying data again from the original host cleared up the errors.

0 Karma

diogofgm
SplunkTrust
SplunkTrust

run a btool to check where is that config coming from. still its weird since you have the config in system/local

/opt/splunk/bin/splunk btool server list --debug disk

Also which Splunk version are you using?

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

nick405060
Motivator

Well 18446744073 petabytes is a lot for just one indexer. I'd spread it out across... two indexers

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...