Deployment Architecture

Data retiring not seems to be working

npandith
Explorer

Currently we are running Splunk server 4.2.3 on a RHEL 5.7 machine and we have set the retirement policy to delete the events which are older than 365 days(31536000 seconds). But when i check the main index for earliest events its giving me earliest event as Nov 5, 2006 3:10:54 PM. My index.conf looks like this-

[default]
maxConcurrentOptimizes = 20
memPoolMB = auto
maxDataSize = auto
defaultDatabase = main
frozenTimePeriodInSecs = 31536000
maxTotalDataSizeMB = 1000000

[main]
maxMemMB = 4096
maxConcurrentOptimizes = 15
maxHotIdleSecs = 1209600
maxDataSize = auto_high_volume
coldPath = /data01/splunk_data/defaultdb/colddb
maxWarmDBCount = 350

[_blocksignature]
homePath = $SPLUNK_DB/blockSignature/db
coldPath = $SPLUNK_DB/blockSignature/colddb
thawedPath = $SPLUNK_DB/blockSignature/thaweddb
maxDataSize = 1000
maxTotalDataSizeMB = 0

[os]
thawedPath = $SPLUNK_DB/os/thaweddb
homePath = $SPLUNK_DB/os/db
coldPath = $SPLUNK_DB/os/colddb
maxHotIdleSecs = 1209600
maxDataSize = auto_high_volume
maxMemMB = 2048

Tags (1)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

You might find this documentation to be helpful:

http://docs.splunk.com/Documentation/Splunk/latest/admin/HowSplunkstoresindexes

Also, what Kristian said is solid advice.

If you want to see what is happening with your buckets, you can look at the 'BucketMover' component of splunkd, which will tell you what is getting moved and why it's been moved. Those messages are logged in $SPLUNK_HOME/var/log/splunk/splunkd.log.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

You might find this documentation to be helpful:

http://docs.splunk.com/Documentation/Splunk/latest/admin/HowSplunkstoresindexes

Also, what Kristian said is solid advice.

If you want to see what is happening with your buckets, you can look at the 'BucketMover' component of splunkd, which will tell you what is getting moved and why it's been moved. Those messages are logged in $SPLUNK_HOME/var/log/splunk/splunkd.log.

kristian_kolb
Ultra Champion

Well, I think the issue here is that data is not retired (i.e. frozen) on a per event basis, but rather on a bucket basis. Only when the newest event in a bucket passes the frozenTimePeriodInSecs value, the bucket as a whole is frozen/retired/deleted. So if you have a bucket in your index that contains data from 2006 and from April 1st 2012, it will not get deleted until April 2nd 2013.

Hope this helps,

Kristian

Drainy
Champion

+1 the Kolbmeister and his description of frozenTimePeriodInSecs

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...