topic Retention Policy of an Index in Knowledge Management

Retention Policy of an Index

Prakash493 — Wed, 30 Sep 2020 00:34:07 GMT

Hi , i am currently setting up the retention policy of an index for data roll over to frozen but i am currently setting data roll over to frozen after 6 months but i am intrested to move data from hot to warm after 2 months and after warm to cold after 3 months then lastly 6 months to frozen curretly below is my setting can you please suggest what modify i need to set for data move from hot to warm and warm to cold.

Indexes.conf
[test]
homePath = $SPLUNK_DB/test/db
ColdPath = $SPLUNK_DB/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
repFactor = auto
coldToFrozenDir = $SPLUNK_DB/test/frozendb
coldToFrozenInSecs = 15552000

So after 6 months my data will roll into frozen , what extra setting i need to made above to make data roll over from hot to warm in 2 months and warm to cold after 2 months and after that will roll into frozen.

Re: Retention Policy of an Index

harsmarvania57 — Wed, 15 May 2019 12:00:32 GMT

Hi,

Let me explain first how splunk roll data from Hot to Warm and from Warm to Cold and from Cold to Frozen. Below scenario explain bucket roll if you are not using volumes.

1.) Hot to Warm Bucket roll depends on 2-3 parameters. First splunk looks for maxDataSize (Default is auto means 750MB) and maxHotSpanSecs (Default is 90 days) in indexes.conf if any of the parameter reach it's limit then bucket will roll from hot to warm. If any of the limit is not reached and hot bucket will be idle then it will roll to warm when maxHotIdleSecs (Default is 90 days) limit will reach.

2.) Warm to Cold buckets roll depends on maxWarmDBCount (Default is 300 warm buckets) when index reaches maxWarmDBCount it will roll from warm to cold.

3.) Cold to Frozen roll depends on maxTotalDataSizeMB (Default is 500000 MB) and frozenTimePeriodInSecs (Default is 6 years) whichever hit first. frozenTimePeriodInSecs applies to all events in single bucket so until and unless all events in bucket are older than frozenTimePeriodInSecs bucket will not be removed.

Based on above parameter you can set these parameter as per your requirement but you will not able to achieve exact rolling of bucket after X months from Hot->Warm->Cold because few of the parameter is depend on number of bucket instead of time and if data is coming to indexer does not parse timestamp properly then you will end up with many buckets.

Re: Retention Policy of an Index

Prakash493 — Wed, 15 May 2019 12:27:27 GMT

Thank you so much for clarification it really helps , if i just want to set my data will roll over to frozen after 6 months so is my above stanza is correct ?

Re: Retention Policy of an Index

harsmarvania57 — Wed, 15 May 2019 12:35:41 GMT

After 6 months do you want to remove data from Splunk or do you want to move bucket to archived directory ?

Re: Retention Policy of an Index

Prakash493 — Wed, 15 May 2019 12:37:09 GMT

Yes please tell for both the cases if i want to archieve what i have to add and if i want to remove what i need to add will be helpfull.

Re: Retention Policy of an Index

harsmarvania57 — Wed, 15 May 2019 12:42:43 GMT

1.) Below configuration to remove data from Splunk after 6 months but this will create many buckets so use with caution because I don't know how much data will come to test index.

indexes.conf

[test]
homePath = $SPLUNK_DB/test/db
ColdPath = $SPLUNK_DB/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
repFactor = auto
maxHotSpanSecs = 86400
frozenTimePeriodInSecs = 5184000

2.) Below configuration to roll data into archived directory, this will also generate many buckets.

indexes.conf

[test]
homePath = $SPLUNK_DB/test/db
ColdPath = $SPLUNK_DB/test/colddb
thawedPath = $SPLUNK_DB/test/thaweddb
repFactor = auto
maxHotSpanSecs = 86400
frozenTimePeriodInSecs = 5184000
coldToFrozenDir = < your archived directory path>
coldToFrozenScript = < your path to script >

Re: Retention Policy of an Index

Prakash493 — Wed, 15 May 2019 12:44:55 GMT

Thank you so much will solve my issue