Getting Data In

Indexes configuration

g_paternicola
Path Finder

Hi everyone,

I'm a bit confused about the retention time of an index. I have created an index (via indexes.conf) with 90 days retention time and max volume of 50GB... so, I always knew, that the logs are gonna delete if the index has reached the max volume or the time has reached 90 days...

But in my case my index has 4.8 GB reached and the oldest event is from the 1st of May, which is more than 90 days...

Do I understand this wrong?

Labels (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Basically You have understood it right. There is still (at least) one exception. When you are starting to collect logs from source systems, usually it gets all old logs too. Those old logs also goes to the same bucket where currents already are. All those are usable as long as that bucket are there. The bucket will be froze after all events inside it has older than your retention time (90d). For that reason there could be some events which are much older than 90d.

r. Ismo

View solution in original post

isoutamo
SplunkTrust
SplunkTrust

Basically You have understood it right. There is still (at least) one exception. When you are starting to collect logs from source systems, usually it gets all old logs too. Those old logs also goes to the same bucket where currents already are. All those are usable as long as that bucket are there. The bucket will be froze after all events inside it has older than your retention time (90d). For that reason there could be some events which are much older than 90d.

r. Ismo

jwalthour
Communicator

I’d need to see your indexes.conf to be sure, but I think you’re talking about frozenTimePeriodInSecs and either maxVolumeDataSizeMB or maxTotalDataSize MB. 

For your sizing, whatever setting you used, if your index size is 4.8 GB out of 50 Gb, you’ve not hit that limits yet to trigger any bucket movement.

For frozenTimePeriodInSecs, this is triggered for index buckets, not individual events. When the youngest event in a bucket is older than 90 days (in your case), then the bucket gets rolled to frozen. However, that means you can have events in an index older than 90 days, like your May 1 event (115 days), if the youngest event in the same bucket it’s in is younger than 90 days. Make sense?

0 Karma

gcusello
Legend

HI @g_paternicola,

retention is managed at bucket level, so when the latest event of a bucket exceeds the retention time, the full bucket is deleted (or moved) not before.

This means that's possible that you have events that exceed the retention time, because they are in a bucket where there events that don't exceed retention time.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...