Getting Data In

Indexes configuration

g_paternicola
Path Finder

Hi everyone,

I'm a bit confused about the retention time of an index. I have created an index (via indexes.conf) with 90 days retention time and max volume of 50GB... so, I always knew, that the logs are gonna delete if the index has reached the max volume or the time has reached 90 days...

But in my case my index has 4.8 GB reached and the oldest event is from the 1st of May, which is more than 90 days...

Do I understand this wrong?

Labels (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Basically You have understood it right. There is still (at least) one exception. When you are starting to collect logs from source systems, usually it gets all old logs too. Those old logs also goes to the same bucket where currents already are. All those are usable as long as that bucket are there. The bucket will be froze after all events inside it has older than your retention time (90d). For that reason there could be some events which are much older than 90d.

r. Ismo

View solution in original post

isoutamo
SplunkTrust
SplunkTrust

Basically You have understood it right. There is still (at least) one exception. When you are starting to collect logs from source systems, usually it gets all old logs too. Those old logs also goes to the same bucket where currents already are. All those are usable as long as that bucket are there. The bucket will be froze after all events inside it has older than your retention time (90d). For that reason there could be some events which are much older than 90d.

r. Ismo

jwalthour
Communicator

I’d need to see your indexes.conf to be sure, but I think you’re talking about frozenTimePeriodInSecs and either maxVolumeDataSizeMB or maxTotalDataSize MB. 

For your sizing, whatever setting you used, if your index size is 4.8 GB out of 50 Gb, you’ve not hit that limits yet to trigger any bucket movement.

For frozenTimePeriodInSecs, this is triggered for index buckets, not individual events. When the youngest event in a bucket is older than 90 days (in your case), then the bucket gets rolled to frozen. However, that means you can have events in an index older than 90 days, like your May 1 event (115 days), if the youngest event in the same bucket it’s in is younger than 90 days. Make sense?

0 Karma

gcusello
SplunkTrust
SplunkTrust

HI @g_paternicola,

retention is managed at bucket level, so when the latest event of a bucket exceeds the retention time, the full bucket is deleted (or moved) not before.

This means that's possible that you have events that exceed the retention time, because they are in a bucket where there events that don't exceed retention time.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...