What do you do if the minimum free disk space is reached on a fresh Splunk installation?

[derejekifle]

I have a fresh install of Splunk on a CentOS VM that has 15GB of disk..
I'm getting the following message... what do I need to do?

1. Dispatch Command: The minimum free disk space (2000MB) reached for /opt/splunk/var/run/splunk/dispatch.
2. Failed to start KV Store process. See mongod.log and splunkd.log for details.
3. Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 347MB, below the minimum of 2000MB. Data writes to index path '/opt/splunk/var/lib/splunk/audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data.
4. KV Store changed status to failed. KVStore process terminated
5. KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

[codebuilder]

Configure logrotate and/or manually purge Splunk log files.

These are unfortunately located at /opt/splunk/var/log/splunk and /opt/splunk/var/log/introspection, which obviously count against your available space on /opt (which is usually small on a standard Linux install).

I generally symlink those directories to /var/log/splunk and /var/log/introspection, with /var/log being on it's own disk, VG, and LV
/dev/mapper/varlogvg01-varloglv01 e.g.

[hardikJsheth]

You are indexing more data than available space. Best would be to increase the disk size, if you are going to keep indexing at same rate.

Just to make it work you can reduce minimum recommended free space to 1GB by logging into your Splunk UI and goto Settings --> General Settings. Change the size for "Pause indexing if free disk space (in MB) falls below ". After changing this values you will have to restart your machine.

For KVStore if it doesn't work after restart check for error message in $SPLUNK_HOME/var/log/splunk/mongod.log.

[tmarlette]

check your free disk space on CentOs under the /opt partition (assuming this is a stand alone instance). You've likely hit your max. if this is machine is a search head and an indexer, you're going to blow through 15GB almost after install.

keep in mind that all default indexes, as well as new indexes default to storing 500GB of data. This means that the index won't rotate out old data until each index hit's 500GB. I'm not sure how many indexes you have, but in your case, 1 is too many at 500GB. adjust the sizes to compensate, and get about 120GB or so on the /opt partition and you should be Ok for awhile.