Monitoring Splunk
Highlighted

Can I change the path of the dispatch directory?

Engager

Hi,

My question is related to this one: http://answers.splunk.com/questions/1140/how-can-i-resolve-the-error-the-minimum-free-disk-space-204...

Namely, I am running into disk space issues. Splunk tries to store searches in /opt/splunk/var/run/splunk/dispatch, but can I change this path? I only have about 1.5 GB on the partition on which Splunk is installed. I set the indexing path to a NFS share with terabytes of storage (/mnt/splunk/var/lib/splunk). I'd like to use this path for search storage as well. Is this possible? Is there a setting in a config file somewhere that I can change?

Thanks! Brian

Tags (2)
Highlighted

Re: Can I change the path of the dispatch directory?

Splunk Employee
Splunk Employee

Nope, you can't specify an alternate location for the dispatch directory.

Options:

  • You could relocate your whole splunk install. If you have less than 1.5GB available, this would be my first suggestion.
  • On Unix, you could try making this directory into a symbolic link. You might discover new misbehaviors, because we don't test this way, but generally Splunk doesn't care about symlinks too much.

Sidenote: Indexing to NFS can be perilous. Be certain your NFS environment is going to work all the time without issues. If it stops operating for any length of time, you'll either get a service interruption, or lost data. I'd rather index to a higher reliability datapath and then quickly migrate the data to NFS, myself.

View solution in original post

Highlighted

Re: Can I change the path of the dispatch directory?

Splunk Employee
Splunk Employee

There is at least one problem still in version 4.1.3 if the dispatch directory is on a (symlinked or sub-mounted) volumed separate from the Splunk config directory, that the outputlookup command will not work. this is scheduled to be fixed in a subsequent maintenance release.

0 Karma
Highlighted

Re: Can I change the path of the dispatch directory?

Path Finder

the issue with the outputlookup command seems to be resolved in 4.1.4:

The "outputlookup" search command doesn't work if var/run/ is on a different volume from etc/apps. (SPL-31765, SPL-31130)

Highlighted

Re: Can I change the path of the dispatch directory?

Splunk Employee
Splunk Employee

There has also recently been at least one instance where the disk space size check did not follow a symlink. Having looked into this, my general consensus is that I wouldn't recommend symlinking as: 1. Most customers don't symlink and your experience will be better the less unique you are. 2. We do create large files and expect fast performance on other paths other than dispatch. 3. Anecdotal evidence exists about disk space size check not following symlinks.

0 Karma
Highlighted

Re: Can I change the path of the dispatch directory?

Contributor

@David - Is this comment still true?

0 Karma
Highlighted

Re: Can I change the path of the dispatch directory?

Engager

I'm running into the same problem running Splunk on AWS (with a small root volume).
Previously moved the indexes to a non root volume. (specified a new splunkdb location in etc/splunk-launch.conf)
I moved the data (don't know if the first step is entirely necessary), symlinked to the external disk and restarted
NOTE: done as the splunk user, and assuming your external volume is mounted at /mnt/data

mkdir /mnt/data/dispatch
mv var/run/splunk/dispatch/* /mnt/data/dispatch/ && rmdir var/run/splunk/dispatch && ln -s /mnt/data/dispatch/ var/run/splunk/dispatch
~/bin/splunk restart

This seems to work for me. There is some report above that the symlink causes problems in previous versions, but I'm not seeing any errors of misbehavior.

Highlighted

Re: Can I change the path of the dispatch directory?

Path Finder

Shockman. Thank you. I had the exact scenario on AWS, added a drive and your solution here of just linking the old output folder with the new worked like a charm!

0 Karma