Getting Data In

How limit my index growth

omuelle1
Communicator

Hi Splunk Users,

I am having an issue with my indexes growing very large and clogging up the space on my disk.

For example: I have noticed the index 'perfmon' getting very large so I went ahead and set the limit to 5 GB. I was reading once the limit is reached it would clean up automically and delete older data. However I see in Fire Brigade that the index size is still 25 GB. How can that be if I limited to be 5 GB?

Thank you,

Oliver

Tags (3)
0 Karma
1 Solution

muebel
SplunkTrust
SplunkTrust

Hi omuelle1, to clarify, I am assuming that you set "maxTotalDataSizeMB" for the index to 5000. If that is the case, some possible explanations:

-Splunk hasn't been restarted and needs to be in order for the change to take effect
-There is something in the way as far as file permissions go, and splunk can't delete the buckets. Check splunkd.log
-Fire Brigade (haven't worked with that) is reporting false information, or old information. do a " du -sh /path/to/index " to find out the current size

Let me know if any of this helps!

View solution in original post

muebel
SplunkTrust
SplunkTrust

Hi omuelle1, to clarify, I am assuming that you set "maxTotalDataSizeMB" for the index to 5000. If that is the case, some possible explanations:

-Splunk hasn't been restarted and needs to be in order for the change to take effect
-There is something in the way as far as file permissions go, and splunk can't delete the buckets. Check splunkd.log
-Fire Brigade (haven't worked with that) is reporting false information, or old information. do a " du -sh /path/to/index " to find out the current size

Let me know if any of this helps!

omuelle1
Communicator

Thank you for the fast answer.

Yes you are right, that's the setting I used.
- I did restart Splunk
- Permissions are fine
- I did check the actual sizes in the folders and it matches what I see in FB

I went ahead and set the retention policy to 5 days
frozenTimePeriodInSecs = 432000

since I really don't need the data longer than 5 days and it actually cleared up space. I might have to clarify that the the index was already 25 GB when I set it to 5 GB max, however I was expecting that it would automatically clean it up to 5 GB.

0 Karma

muebel
SplunkTrust
SplunkTrust

Hi omulle1, glad to help! 😄

Yes, there is an interval ( I think 60 seconds by default ) that splunk will examine it's indexes and freeze buckets ( individual folders within an index directory ) based on the configuration in indexes.conf.

0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...