Getting Data In

buckets - Frozen and Thawed bucket

VijaySrrie
Builder

Hi,

As soon as data moves from cold to frozen bucket it gets deleted? How data moves from frozen bucket to Thawed bucket. The data in thawed bucket is that searchable? How long data will be in thawed bucket? will that move back to frozen bucket again?

If we need the data for years where and how to store it?

Labels (1)
Tags (1)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Default behavior for rolling from Cold to Frozen is to delete the bucket. Once a roll to frozen script is configured, the bucketroller process will run the script and data will be moved from the index to the frozen volume.

To get data into thawed, you will need to automate a process, or manually copy the data to the defined thawedPath for the index in indexes.conf. Data in the thawedPath is not managed by lifecycle policies. So once the data is moved into thawed, you will need to delete it once you're done searching it and using it.

For storing data long term, there are some things to consider. First would be how long does your data need to be searchable? 3 months? 6 months? 1 year? 3 years? The answer to this is obviously going to effect your hot/warm, cold, and frozen sizing. Smartstore utilizing S3/Object storage helps reduce this cost for long term searchable storage. But if you're not able to utilize this, then you have to make some decisions around how much money you for hardware.

For legacy deployments, most customers will have various indexes that have different term requirements for compliance. Typically anything searchable over 1 year isn't done. What most customers will typically do is store long term frozen data on cheaper storage in SAN. Then the restore process for after 1 year is part of an operational request.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...