Free disk space (5000MB) reached solved but storage did not reduce

[wuming79]

Hi,
I just set my retirement policy due to space issue (reference: https://answers.splunk.com/answers/583891/which-indexesconf-should-i-edit-to-set-retirement.html)

My vm used storage is the same before and after I set my retirement policy. Does setting retirement policy to delete anything that is more than 1 month old actually helps to reduce the used storage space?

What files can I delete to reduce the storage space so that I can reduced my Provisional Storage?

[jkat54]

Your index buckets are in

 /opt/splunk/var/lib/splunk/indexName/db

By default. You can delete some of those buckets but before you do, look at the current size of the buckets in each index. You might find you’ve got 1 year of data in just one bucket, or a number of other things you didn’t know was happening.

My assumption is that you have some hot or warm buckets with a lot of data in them and changing your settings didn’t affect these buckets that already existed and they contain events younger than whatever frozenTimePeriodInSecs you specified.

Hope this helps!

[wuming79]

indexName folder is not in /opt/splunk/var/lib/splunk/.
I tried locate IndexName and terminal returns nothing.

[gcusello]

Hi wuming79,
to reduce indexes dimensions you can use two ways:

• reduce retention time,
• reduce indexes dimensions.

in first case, you have to modify indexes.conf adding the frozenTimePeriodInSecs = xxx row and restart Splunk .
In the second case, you can do this by web interface without Splunk restart.

Remember that if you use retention by time, you could have events older that retention period because events are deleted only when the earliest event of the bucket is out of retention period.

So, think to a capacity planning with regards with your monitoring requirements before do something!

Bye.
Giuseppe

[wuming79]

I can't seem to find the indexName folder.
using ncdu on the folder saw 864MB in kvstore. The rest are only in KB.

[jkat54]

Please provide your indexes.conf settings as shown by this command

 /opt/splunk/bin/splunk btool indexes list —debug

That’s hyphen hyphen - - no space

[wuming79]

Hi, it says splunk command not found.

[gcusello]

sorry I forgot a word, use

/opt/splunk/bin/splunk cmd btool indexes list —debug

Bye.
Giuseppe

[jkat54]

cmd isn’t required with the btool command. There must have been another typo or different path to the splunk binary

[wuming79]

Hi what can I do now to reduce the storage capacity? Will deleting all files in dispatch folder help?

[gcusello]

yes it helps but is a temporary solution because dispatch folder files will be ricreated in a few time.
if you can, the best way is to reduce indexes dimensions.
Think to reduce internal indexes (especially _internal) that usually are forgotten.
Bye.
Giuseppe