Getting Data In

buckets - Frozen and Thawed bucket

VijaySrrie
Builder

Hi,

As soon as data moves from cold to frozen bucket it gets deleted? How data moves from frozen bucket to Thawed bucket. The data in thawed bucket is that searchable? How long data will be in thawed bucket? will that move back to frozen bucket again?

If we need the data for years where and how to store it?

Tags (1)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

0 Karma
Get Updates on the Splunk Community!

Unlock Database Monitoring with Splunk Observability Cloud

  In today’s fast-paced digital landscape, even minor database slowdowns can disrupt user experiences and ...

Purpose in Action: How Splunk Is Helping Power an Inclusive Future for All

At Cisco, purpose isn’t a tagline—it’s a commitment. Cisco’s FY25 Purpose Report outlines how the company is ...

[Upcoming Webinar] Demo Day: Transforming IT Operations with Splunk

Join us for a live Demo Day at the Cisco Store on January 21st 10:00am - 11:00am PST In the fast-paced world ...