Getting Data In

buckets - Frozen and Thawed bucket

VijaySrrie
Builder

Hi,

As soon as data moves from cold to frozen bucket it gets deleted? How data moves from frozen bucket to Thawed bucket. The data in thawed bucket is that searchable? How long data will be in thawed bucket? will that move back to frozen bucket again?

If we need the data for years where and how to store it?

Labels (1)
Tags (1)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...