Indexes configuration

[g_paternicola]

Hi everyone,

I'm a bit confused about the retention time of an index. I have created an index (via indexes.conf) with 90 days retention time and max volume of 50GB... so, I always knew, that the logs are gonna delete if the index has reached the max volume or the time has reached 90 days...

But in my case my index has 4.8 GB reached and the oldest event is from the 1st of May, which is more than 90 days...

Do I understand this wrong?

[isoutamo · Accepted solution]

Basically You have understood it right. There is still (at least) one exception. When you are starting to collect logs from source systems, usually it gets all old logs too. Those old logs also goes to the same bucket where currents already are. All those are usable as long as that bucket are there. The bucket will be froze after all events inside it has older than your retention time (90d). For that reason there could be some events which are much older than 90d.

r. Ismo

[jwalthour]

I’d need to see your indexes.conf to be sure, but I think you’re talking about frozenTimePeriodInSecs and either maxVolumeDataSizeMB or maxTotalDataSize MB. 

For your sizing, whatever setting you used, if your index size is 4.8 GB out of 50 Gb, you’ve not hit that limits yet to trigger any bucket movement.

For frozenTimePeriodInSecs, this is triggered for index buckets, not individual events. When the youngest event in a bucket is older than 90 days (in your case), then the bucket gets rolled to frozen. However, that means you can have events in an index older than 90 days, like your May 1 event (115 days), if the youngest event in the same bucket it’s in is younger than 90 days. Make sense?

[gcusello]

HI @g_paternicola,

retention is managed at bucket level, so when the latest event of a bucket exceeds the retention time, the full bucket is deleted (or moved) not before.

This means that's possible that you have events that exceed the retention time, because they are in a bucket where there events that don't exceed retention time.

Ciao.

Giuseppe