If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it?

[jonathanfalconi]

Hi,

If the maxTotalDataSizeMB is reached, how much data will splunk remove from the index and will it freeze it or simply delete?

Lastly, what can you search for in the splunkd logs to indicate the max size was reached?

thanks.

[somesoni2]

This is a good read on Splunk data retention policy implementation.

http://wiki.splunk.com/Deploy:BucketRotationAndRetention

[lguinn2 · Accepted solution]

If the max size is reached, Splunk will freeze the oldest bucket in the index to make room for the new data. if you have not specified a cold-to-frozen directory or script, then Splunk will delete the oldest bucket instead. Splunk will only remove sufficient buckets to reduce the index size beneath the maximum, so that is probably just one bucket - but it may repeat this action frequently as new data arrives.

I played around with this a bit, and I think this will tell you what's happening as buckets are frozen:

index=_internal sourcetype=splunkd yourindexname component=BucketMover

[jonathanfalconi]

grep'ing splunkd.log for 'bucketmover' I was able to find when and which buckets were frozen - thank you.