filesystem full - now what?

[a212830]

Hi,

My splunk search-head/indexer filled up the filesystem that it was running on. When I try to login, it give me an error (out of space). I wanted to reduce the size of certain indexers, to clean-up space. How can I do that without the gui? Anything else that I can delete?

[lguinn2]

+1 to Martin's answer. Also, since you probably need to recover at least a little disk space for Splunk to restart -
You can examine the contents of the following directories and delete files:

Splunk's own logs: $SPLUNK_HOME/var/log/splunk

Search results for running searches, and saved search results: $SPLUNK_HOME/var/run/splunk/dispatch

Note that removing the saved search results may cause some users to need to re-run old searches; in some environments, this might not be a good idea.

Finally, you might think about setting your maximum index sizes so that the sum of all indexes cannot exceed your disk capacity.

[martin_mueller]

You can always edit indexes.conf to reduce index sizes, no UI required.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

[I_am_Jeff]

My version in 4.3. I've used maxDataSizeMB option to limit sizes. If you are using different file systems for hot and cold indexes, as I am, homePath.maxDataSizeMB and coldPath.masDataSizeMB operate independently. My experience is it takes Splunk a while to clean house once you set these options and restart.