topic Re: Roll hotwarm data to cold when frozenTimePeriodInSecs is met in Deployment Architecture

Roll hotwarm data to cold when frozenTimePeriodInSecs is met

rickymckenzie10 — Tue, 12 Mar 2024 22:54:28 GMT

Hello, how can I ensure the data being sent to cool_index is rolled to cold when the data is 120 days old?

The config I'll use

[cool_index] homePath = volume:hotwarm/cool_index/db coldPath = volume:cold/cool_index/colddb thawedPath = $SPLUNK_DB/cool_index/thaweddb frozenTimePeriodInSecs = 10368000 #120 day retention maxTotalDataSizeMB = 60000 maxDataSize=auto repFactor=auto

am I missing something?

Re: Roll hotwarm data to cold when frozenTimePeriodInSecs is met

kiran_panchavat — Wed, 13 Mar 2024 05:52:28 GMT

@rickymckenzie10 I think that you should read at https://docs.splunk.com/Documentation/Splunk/9.2.0/Admin/Indexesconf

https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

frozenTimePeriodInSecs = <nonnegative integer>
* The number of seconds after which indexed data rolls to frozen.
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
frozen.
* NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs'
seconds before the bucket rolls to frozen.
* The highest legal value is 4294967295.
* Default: 188697600 (6 years)

maxTotalDataSizeMB = <nonnegative integer>
* The maximum size of an index, in megabytes.
* If an index grows larger than the maximum size, splunkd freezes the oldest
data in the index.
* This setting applies only to hot, warm, and cold buckets. It does
not apply to thawed buckets.
* CAUTION: The 'maxTotalDataSizeMB' size limit can be reached before the time 
limit defined in 'frozenTimePeriodInSecs' due to the way bucket time spans 
are calculated. When the 'maxTotalDataSizeMB' limit is reached, the buckets 
are rolled to frozen. As the default policy for frozen data is deletion, 
unintended data loss could occur.
* Splunkd ignores this setting on remote storage enabled indexes.
* Highest legal value is 4294967295
* Default: 500000

Re: Roll hotwarm data to cold when frozenTimePeriodInSecs is met

isoutamo — Wed, 13 Mar 2024 13:38:34 GMT

that .conf presentation which @kiran_panchavat are referring is excellent even it's little bit old and don't contains all new stuff like S2 (Splunk Smart Store). Please read it and also some other answers which are talking bout that same issue.

Shortly, You cannot ensure that events are moved into cold storage based on age! There are no parameter which define this for warm bucket. Moving warm to cold is defined base on bucket count not based on time.

frozenTimePerioInSecs is used for moving cold buckets to frozen (archiving those outside of splunk or remove those as default action).

r. Ismo