https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174331#M23323 <P>Hi yannk, thanks for the answer - very useful and I think I have a handle on it now in terms of the various states. I just have one more question - if we were to set maxTotalDataSizeMB and frozenTimePeriodInSecs to ensure that we don't keep too much data in active indexes, but due to the default settings within indexes.conf meant that data had not moved from WARM to COLD would that data be moved from WARM to FROZEN directly?</P> Fri, 08 May 2015 10:10:22 GMT john_howley 2015-05-08T10:10:22Z https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174329#M23321 <P>Can I automatically set a limit that keeps only say the last month of indexed data for example with nmon data. Or do I just delete the directories directly in the filesystem?</P> Thu, 07 May 2015 14:43:00 GMT https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174329#M23321 john_howley 2015-05-07T14:43:00Z https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174330#M23322 <P>Read indexes.conf specifications. You will need to edit a config file.<BR /> see <A href="http://docs.splunk.com/Documentation/Splunk/6.2.3/Indexer/Setaretirementandarchivingpolicy">http://docs.splunk.com/Documentation/Splunk/6.2.3/Indexer/Setaretirementandarchivingpolicy</A></P> <P>setup a ** frozenTimePeriodInSecs** to be the day limit. (default 6 years)</P> <P>and if you retention is very low (less than months) , you may also want to reduce the <STRONG>maxHotSpanSecs</STRONG> (maximum hot bucket timespan default to 90 days) to force them to rotate more often. (as hot buckets cannot be frozen)</P> <P>Remark : do not use exactly 1 hour or 1 day for maxHotSpanSecs, It's best to use one week, it prevent too many buckets to be created if your events are not received chronologically.</P> Thu, 07 May 2015 15:19:21 GMT https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174330#M23322 yannK 2015-05-07T15:19:21Z https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174331#M23323 <P>Hi yannk, thanks for the answer - very useful and I think I have a handle on it now in terms of the various states. I just have one more question - if we were to set maxTotalDataSizeMB and frozenTimePeriodInSecs to ensure that we don't keep too much data in active indexes, but due to the default settings within indexes.conf meant that data had not moved from WARM to COLD would that data be moved from WARM to FROZEN directly?</P> Fri, 08 May 2015 10:10:22 GMT https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174331#M23323 john_howley 2015-05-08T10:10:22Z https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174332#M23324 <P>the lifetime of a bucket is :</P> <OL> <LI><STRONG>hot</STRONG> (read and write) - on the homePath</LI> <LI>then <STRONG>warm</STRONG> or <STRONG>cold</STRONG> - on the homePath or coldPath respectively</LI> <LI>then <STRONG>frozen</STRONG> (deleted or archived is a coldtoFrozenDir or script is defined) </LI> <LI>and eventually <STRONG>thawed</STRONG> if restored from frozen - in the thawedPath</LI> </OL> <P>So a bucket can go directly from warm to frozen without being cold.<BR /> The Cold state is optional, you may end up with an empty coldPath with the splunk defaults. And is only useful if you are using homePath and coldPath on different partitions for space reasons.</P> Fri, 08 May 2015 16:02:38 GMT https://community.splunk.com/t5/Deployment-Architecture/limit-number-of-days-of-index-data-kept/m-p/174332#M23324 yannK 2015-05-08T16:02:38Z