Data retiring not seems to be working

[npandith]

Currently we are running Splunk server 4.2.3 on a RHEL 5.7 machine and we have set the retirement policy to delete the events which are older than 365 days(31536000 seconds). But when i check the main index for earliest events its giving me earliest event as Nov 5, 2006 3:10:54 PM. My index.conf looks like this-

[default]
maxConcurrentOptimizes = 20
memPoolMB = auto
maxDataSize = auto
defaultDatabase = main
frozenTimePeriodInSecs = 31536000
maxTotalDataSizeMB = 1000000

[main]
maxMemMB = 4096
maxConcurrentOptimizes = 15
maxHotIdleSecs = 1209600
maxDataSize = auto_high_volume
coldPath = /data01/splunk_data/defaultdb/colddb
maxWarmDBCount = 350

[_blocksignature]
homePath = $SPLUNK_DB/blockSignature/db
coldPath = $SPLUNK_DB/blockSignature/colddb
thawedPath = $SPLUNK_DB/blockSignature/thaweddb
maxDataSize = 1000
maxTotalDataSizeMB = 0

[os]
thawedPath = $SPLUNK_DB/os/thaweddb
homePath = $SPLUNK_DB/os/db
coldPath = $SPLUNK_DB/os/colddb
maxHotIdleSecs = 1209600
maxDataSize = auto_high_volume
maxMemMB = 2048

[jbsplunk · Accepted solution]

You might find this documentation to be helpful:

http://docs.splunk.com/Documentation/Splunk/latest/admin/HowSplunkstoresindexes

Also, what Kristian said is solid advice.

If you want to see what is happening with your buckets, you can look at the 'BucketMover' component of splunkd, which will tell you what is getting moved and why it's been moved. Those messages are logged in $SPLUNK_HOME/var/log/splunk/splunkd.log.

[kristian_kolb]

Well, I think the issue here is that data is not retired (i.e. frozen) on a per event basis, but rather on a bucket basis. Only when the newest event in a bucket passes the frozenTimePeriodInSecs value, the bucket as a whole is frozen/retired/deleted. So if you have a bucket in your index that contains data from 2006 and from April 1st 2012, it will not get deleted until April 2nd 2013.

Hope this helps,

Kristian

[Drainy]

+1 the Kolbmeister and his description of frozenTimePeriodInSecs