topic Re: If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it? in Monitoring Splunk

If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it?

jonathanfalconi — Mon, 01 Dec 2014 00:19:41 GMT

Hi,

If the maxTotalDataSizeMB is reached, how much data will splunk remove from the index and will it freeze it or simply delete?

Lastly, what can you search for in the splunkd logs to indicate the max size was reached?

thanks.

Re: If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it?

lguinn2 — Mon, 01 Dec 2014 00:57:40 GMT

If the max size is reached, Splunk will freeze the oldest bucket in the index to make room for the new data. if you have not specified a cold-to-frozen directory or script, then Splunk will delete the oldest bucket instead. Splunk will only remove sufficient buckets to reduce the index size beneath the maximum, so that is probably just one bucket - but it may repeat this action frequently as new data arrives.

I played around with this a bit, and I think this will tell you what's happening as buckets are frozen:

index=_internal sourcetype=splunkd yourindexname component=BucketMover

Re: If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it?

somesoni2 — Mon, 01 Dec 2014 06:05:19 GMT

This is a good read on Splunk data retention policy implementation.

http://wiki.splunk.com/Deploy:BucketRotationAndRetention

Re: If maxTotalDataSizeMB is reached, how much data will Splunk remove from the index and will it freeze or delete it?

jonathanfalconi — Thu, 04 Dec 2014 13:23:46 GMT

grep'ing splunkd.log for 'bucketmover' I was able to find when and which buckets were frozen - thank you.