topic Re: How to delete raw data files? in Getting Data In

How to delete raw data files?

stwong — Mon, 28 Dec 2015 09:56:15 GMT

Hi all,

We encounter Splunk server running out of disk space issue in past months. I tried to reduce maxTotalDataSizeMB and frozenTimePeriodInSecs from time to time for squeezing disk space. Currently it looks like the following:

------------ cut here ---------------
[main]

maxTotalDataSizeMB = 20000
coldToFrozenDir = /usr/local/splunk/var/frozen/main
frozenTimePeriodInSecs = 864000

[p0f]
maxTotalDataSizeMB = 20000
coldToFrozenDir = /usr/local/splunk/var/frozen/p0f
frozenTimePeriodInSecs = 432000

------------ cut here ---------------

Seems default of frozen data processing is to delete them. However, raw data is still there and eats up a lot of disk space, e.g. in /usr/local/splunk/var/frozen/p0f for the p0f index, there are log of db_* folders:

ls -l db_1415393077_1415384845_1994/rawdata/
total 112132
-rw-------. 1 splunk splunk 114815328 Nov  8  2014 journal.gz

I daren’t deleting them manually.

Would anyone please help? Sorry for the newbie question.

Thanks and regards
/ST Wong

Re: How to delete raw data files?

richgalloway — Mon, 28 Dec 2015 13:47:33 GMT

Yes, the default action is to delete frozen data. However, by specifying a value for coldToFrozenDir you have changed the default behaviour so Splunk will retain frozen data. Remove that attribute, restart splunkd and the frozen data will be deleted.

Re: How to delete raw data files?

renjith_nair — Tue, 29 Sep 2020 08:13:11 GMT

If you set the coldToFrozenDir attribute in indexes.conf, the indexer will automatically copy frozen buckets to the specified location before erasing the data from the index. So the data still resides on the location you specified on the disk.

If you don't specify either of these attributes(coldToFrozenDir or coldToFrozenScript), the indexer runs a default script that simply writes the name of the bucket being erased to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log. It then erases the bucket.

Reference : http://docs.splunk.com/Documentation/Splunk/6.3.1511/Indexer/Automatearchiving

Re: How to delete raw data files?

stwong — Tue, 29 Dec 2015 01:05:18 GMT

Hi all,

Thanks for your help. It works.

Best Regards

Re: How to delete raw data files?

renjith_nair — Tue, 29 Dec 2015 05:16:27 GMT

Please accept this as answer if you are happy so that question will be closed and might be useful for others.

Re: How to delete raw data files?

ppablo — Tue, 05 Jan 2016 22:19:41 GMT

Hi @stwong

Please be sure to resolve your posts by clicking Accept directly below the answer that best answered your question. That will make this post easier to find for other users with the same/similar question.

Re: How to delete raw data files?

stwong — Wed, 06 Jan 2016 01:01:54 GMT

Noted and thanks. Seems can only accept the "best" one even I find all the replies are helpful...
Thanks for your reminder.