Getting Data In

buckets - Frozen and Thawed bucket

VijaySrrie
Builder

Hi,

As soon as data moves from cold to frozen bucket it gets deleted? How data moves from frozen bucket to Thawed bucket. The data in thawed bucket is that searchable? How long data will be in thawed bucket? will that move back to frozen bucket again?

If we need the data for years where and how to store it?

Labels (1)
Tags (1)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...