Getting Data In

Free disk space (5000MB) reached solved but storage did not reduce

wuming79
Path Finder

Hi,
I just set my retirement policy due to space issue (reference: https://answers.splunk.com/answers/583891/which-indexesconf-should-i-edit-to-set-retirement.html)

My vm used storage is the same before and after I set my retirement policy. Does setting retirement policy to delete anything that is more than 1 month old actually helps to reduce the used storage space?

What files can I delete to reduce the storage space so that I can reduced my Provisional Storage?

alt text

0 Karma

jkat54
SplunkTrust
SplunkTrust

Your index buckets are in

 /opt/splunk/var/lib/splunk/indexName/db

By default. You can delete some of those buckets but before you do, look at the current size of the buckets in each index. You might find you’ve got 1 year of data in just one bucket, or a number of other things you didn’t know was happening.

My assumption is that you have some hot or warm buckets with a lot of data in them and changing your settings didn’t affect these buckets that already existed and they contain events younger than whatever frozenTimePeriodInSecs you specified.

Hope this helps!

0 Karma

wuming79
Path Finder

indexName folder is not in /opt/splunk/var/lib/splunk/.
I tried locate IndexName and terminal returns nothing.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi wuming79,
to reduce indexes dimensions you can use two ways:

  • reduce retention time,
  • reduce indexes dimensions.

in first case, you have to modify indexes.conf adding the frozenTimePeriodInSecs = xxx row and restart Splunk .
In the second case, you can do this by web interface without Splunk restart.

Remember that if you use retention by time, you could have events older that retention period because events are deleted only when the earliest event of the bucket is out of retention period.

So, think to a capacity planning with regards with your monitoring requirements before do something!

Bye.
Giuseppe

0 Karma

wuming79
Path Finder

I can't seem to find the indexName folder.
using ncdu on the folder saw 864MB in kvstore. The rest are only in KB.

alt text

0 Karma

jkat54
SplunkTrust
SplunkTrust

Please provide your indexes.conf settings as shown by this command

 /opt/splunk/bin/splunk btool indexes list —debug

That’s hyphen hyphen - - no space

0 Karma

wuming79
Path Finder

Hi, it says splunk command not found.

alt text

0 Karma

gcusello
SplunkTrust
SplunkTrust

sorry I forgot a word, use

/opt/splunk/bin/splunk cmd btool indexes list —debug

Bye.
Giuseppe

0 Karma

jkat54
SplunkTrust
SplunkTrust

cmd isn’t required with the btool command. There must have been another typo or different path to the splunk binary

0 Karma

wuming79
Path Finder

Hi what can I do now to reduce the storage capacity? Will deleting all files in dispatch folder help?

0 Karma

gcusello
SplunkTrust
SplunkTrust

yes it helps but is a temporary solution because dispatch folder files will be ricreated in a few time.
if you can, the best way is to reduce indexes dimensions.
Think to reduce internal indexes (especially _internal) that usually are forgotten.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...