Hi,
My question is related to this one: http://answers.splunk.com/questions/1140/how-can-i-resolve-the-error-the-minimum-free-disk-space-204...
Namely, I am running into disk space issues. Splunk tries to store searches in /opt/splunk/var/run/splunk/dispatch, but can I change this path? I only have about 1.5 GB on the partition on which Splunk is installed. I set the indexing path to a NFS share with terabytes of storage (/mnt/splunk/var/lib/splunk). I'd like to use this path for search storage as well. Is this possible? Is there a setting in a config file somewhere that I can change?
Thanks! Brian
Nope, you can't specify an alternate location for the dispatch directory.
Options:
Sidenote: Indexing to NFS can be perilous. Be certain your NFS environment is going to work all the time without issues. If it stops operating for any length of time, you'll either get a service interruption, or lost data. I'd rather index to a higher reliability datapath and then quickly migrate the data to NFS, myself.
I'm running into the same problem running Splunk on AWS (with a small root volume).
Previously moved the indexes to a non root volume. (specified a new splunkdb location in etc/splunk-launch.conf)
I moved the data (don't know if the first step is entirely necessary), symlinked to the external disk and restarted
NOTE: done as the splunk user, and assuming your external volume is mounted at /mnt/data
mkdir /mnt/data/dispatch
mv var/run/splunk/dispatch/* /mnt/data/dispatch/ && rmdir var/run/splunk/dispatch && ln -s /mnt/data/dispatch/ var/run/splunk/dispatch
~/bin/splunk restart
This seems to work for me. There is some report above that the symlink causes problems in previous versions, but I'm not seeing any errors of misbehavior.
Shockman. Thank you. I had the exact scenario on AWS, added a drive and your solution here of just linking the old output folder with the new worked like a charm!
Nope, you can't specify an alternate location for the dispatch directory.
Options:
Sidenote: Indexing to NFS can be perilous. Be certain your NFS environment is going to work all the time without issues. If it stops operating for any length of time, you'll either get a service interruption, or lost data. I'd rather index to a higher reliability datapath and then quickly migrate the data to NFS, myself.
There has also recently been at least one instance where the disk space size check did not follow a symlink. Having looked into this, my general consensus is that I wouldn't recommend symlinking as: 1. Most customers don't symlink and your experience will be better the less unique you are. 2. We do create large files and expect fast performance on other paths other than dispatch. 3. Anecdotal evidence exists about disk space size check not following symlinks.
@David - Is this comment still true?
the issue with the outputlookup command seems to be resolved in 4.1.4:
The "outputlookup" search command doesn't work if var/run/ is on a different volume from etc/apps. (SPL-31765, SPL-31130)
There is at least one problem still in version 4.1.3 if the dispatch directory is on a (symlinked or sub-mounted) volumed separate from the Splunk config directory, that the outputlookup
command will not work. this is scheduled to be fixed in a subsequent maintenance release.