topic Re: Retention policy for index not working as intended in Getting Data In

Retention policy for index not working as intended

annebeate — Mon, 28 Sep 2020 15:47:58 GMT

Hi,

I have a index which should only have data for the past 3 hours. I've set the frozenTimePeriodInSecs to 10800. I activated the configuration on the 2nd of February, but data are still present from that date. I think I need to reduce the size of the buckets. Could you please recommend which parameters I should set to accomplish a retention policy of 3 hours? The size of the index is 28 MB as of now.

[s02683_minesider_prod_audit]
coldPath = $SPLUNK_DB/s02683_minesider_prod_audit/colddb
homePath = $SPLUNK_DB/s02683_minesider_prod_audit/db
thawedPath = $SPLUNK_DB/s02683_minesider_prod_audit/thaweddb
frozenTimePeriodInSecs = 10800

Re: Retention policy for index not working as intended

kristian_kolb — Tue, 04 Feb 2014 09:23:06 GMT

Hi,

You know that splunk will only freeze (i.e. delete) a bucket when the newest event in that bucket is older than your retention limit. Unless you have very high volume of traffic, you need to set your bucket size rather small.

For example, if your index receives 60 MB of logs per hour, you could set your bucket size to 10 MB. With average compression rates (~50%) you should have about 20 minutes worth of log data per bucket (as long as you only have one hot bucket at a time). Given that you only freeze when the newest event is too old, the oldest events in your index should be 3h20m at any given time.

However, working with so small indexes and buckets is not what Splunk was engineered for, and I don't know how often freeze checks are actually made. Also, I think that from a performance perspective, larger buckets (>750 MB) are more efficient, but then again, your data set seems rather small, so that perhaps has less impact in your case.

Re: Retention policy for index not working as intended

annebeate — Wed, 05 Feb 2014 07:51:41 GMT

Thanks for your reply 🙂 Which parameter do you recommend that I use: maxHotSpanSecs or homePath.maxDataSizeMB?

homePath.maxDataSizeMB =
* Limits the size of the hot/warm DB to the maximum specified size, in MB.

maxHotSpanSecs =
* Upper bound of timespan of hot/warm buckets in seconds.
* Defaults to 7776000 seconds (90 days).

Re: Retention policy for index not working as intended

kristian_kolb — Wed, 05 Feb 2014 12:18:01 GMT

I was more thinking about maxDataSize.
maxHotSpanSecs might work too, but I think I would prefer the combination of frozenTimePeriodInSecs and maxDataSize. I have not played around with this extensively, so do not take my advice as Divine Truth.

Oh, and the freeze checks are controlled through the rotatePeriodInSecs parameter.