My filesystem is full and splunk stopped indexing, how do i make space/start splunk

[Chris_R_]

My filesystem is full and splunk wont start.
How do i make some last minute filesystem space and start splunk?
What are some things i can do to avoid this in the future?

[the_wolverine]

There are 2 files in $SPLUNK_HOME/var/log/splunk/ that can grow without bounds. This is a bug. These are the web_access.log and web_service.log. You may find that these files are over 1GB+ in size. If you're not in the middle of troubleshooting an issue that requires these files, simply delete them. Or, if you need the latest content, do the following:

• cd to $SPLUNK_HOME/var/log/splunk/
• tail -10000 web_access.log > web_access.log.1
• rm web_access.log
• repeat if needed for web_service.log

As Chris R mentions, if you're running on Windows you'll want to check for gargantuan *.DMP files in $SPLUNK_HOME/var/log/splunk/ which can also be over 1GB in size.

If this is unix, try checking for recent core files created that have filled up your disk:

find / -type f -name core

[Chris_R_ · Accepted solution]

Until the crisis is resolved you have a few options:
If your storing your colddb on the same filesystem as your hot/warm buckets you can temporarily move the colddb to another filesystem.
These are the dirs named
$SPLUNK_HOME/var/lib/splunk/index_name/colddb/db_##########_##########_##
Once that is done edit your $SPLUNK_HOME/etc/system/local/indexes.conf so splunk can find it.

[index_name]
coldPath = /some_other_filesystem/index_name/colddb

Also you can temporarily move some of your warm buckets to another filesystem/backup device.
These buckets will be unavailable for searches once spunk starts but you will at least be back up and running until you can make more space.
Once you resolve the filesystem space problem you can stop splunk/move them back/start splunk.
These are the dirs named
SPLUNK_HOME/var/lib/splunk/index_name/db/db_##########_##########_##

On windows? delete old / large .dmp files in(these contain crash info)
$SPLUNK_HOME/var/log

Avoid this in the future by regularly checking your Status->Index activity page as well as monitoring how much data is coming in from your forwarders with this search
index=_internal per_host_thruput | eval gb=((kb/1024) /1024) | timechart span=1d sum(gb) by series | addtotals

More bucket info at: http://www.splunk.com/wiki/Deploy:UnderstandingBuckets