topic Re: Free disk space (5000MB) reached solved but storage did not reduce in Getting Data In

Free disk space (5000MB) reached solved but storage did not reduce

wuming79 — Thu, 19 Oct 2017 09:09:35 GMT

Hi,
I just set my retirement policy due to space issue (reference: https://answers.splunk.com/answers/583891/which-indexesconf-should-i-edit-to-set-retirement.html)

My vm used storage is the same before and after I set my retirement policy. Does setting retirement policy to delete anything that is more than 1 month old actually helps to reduce the used storage space?

What files can I delete to reduce the storage space so that I can reduced my Provisional Storage?

Re: Free disk space (5000MB) reached solved but storage did not reduce

gcusello — Thu, 19 Oct 2017 09:21:21 GMT

Hi wuming79,
to reduce indexes dimensions you can use two ways:

reduce retention time,
reduce indexes dimensions.

in first case, you have to modify indexes.conf adding the frozenTimePeriodInSecs = xxx row and restart Splunk .
In the second case, you can do this by web interface without Splunk restart.

Remember that if you use retention by time, you could have events older that retention period because events are deleted only when the earliest event of the bucket is out of retention period.

So, think to a capacity planning with regards with your monitoring requirements before do something!

Bye.
Giuseppe

Re: Free disk space (5000MB) reached solved but storage did not reduce

wuming79 — Thu, 19 Oct 2017 09:27:49 GMT

Hi what can I do now to reduce the storage capacity? Will deleting all files in dispatch folder help?

Re: Free disk space (5000MB) reached solved but storage did not reduce

jkat54 — Thu, 19 Oct 2017 10:00:41 GMT

Your index buckets are in

 /opt/splunk/var/lib/splunk/indexName/db

By default. You can delete some of those buckets but before you do, look at the current size of the buckets in each index. You might find you’ve got 1 year of data in just one bucket, or a number of other things you didn’t know was happening.

My assumption is that you have some hot or warm buckets with a lot of data in them and changing your settings didn’t affect these buckets that already existed and they contain events younger than whatever frozenTimePeriodInSecs you specified.

Hope this helps!

Re: Free disk space (5000MB) reached solved but storage did not reduce

gcusello — Thu, 19 Oct 2017 11:08:35 GMT

yes it helps but is a temporary solution because dispatch folder files will be ricreated in a few time.
if you can, the best way is to reduce indexes dimensions.
Think to reduce internal indexes (especially _internal) that usually are forgotten.
Bye.
Giuseppe

Re: Free disk space (5000MB) reached solved but storage did not reduce

wuming79 — Thu, 19 Oct 2017 14:39:24 GMT

I can't seem to find the indexName folder.
using ncdu on the folder saw 864MB in kvstore. The rest are only in KB.

Re: Free disk space (5000MB) reached solved but storage did not reduce

jkat54 — Thu, 19 Oct 2017 22:09:53 GMT

Please provide your indexes.conf settings as shown by this command

 /opt/splunk/bin/splunk btool indexes list —debug

That’s hyphen hyphen - - no space

Re: Free disk space (5000MB) reached solved but storage did not reduce

wuming79 — Fri, 20 Oct 2017 02:14:52 GMT

Hi, it says splunk command not found.

Re: Free disk space (5000MB) reached solved but storage did not reduce

gcusello — Fri, 20 Oct 2017 12:00:19 GMT

sorry I forgot a word, use

/opt/splunk/bin/splunk cmd btool indexes list —debug

Bye.
Giuseppe

Re: Free disk space (5000MB) reached solved but storage did not reduce

jkat54 — Fri, 20 Oct 2017 18:57:27 GMT

cmd isn’t required with the btool command. There must have been another typo or different path to the splunk binary

Re: Free disk space (5000MB) reached solved but storage did not reduce

wuming79 — Mon, 23 Oct 2017 01:33:40 GMT

indexName folder is not in /opt/splunk/var/lib/splunk/.
I tried locate IndexName and terminal returns nothing.