Deployment Architecture

Hot/Warm and Cold Storage

panpanbebe
New Member

We are in the process increase our daily ingest rate to 2TB, and I want to ask the questions about our storage retention policy design. The hot/warm/cold can be searchable from Splunk, what's the ideal retention for cold storage? my contractor design the same period which I am a little confused. thank you

 

Hot/Warm: 90 days

Cold: 90 days ( 

Archive: 3 years

Labels (1)
0 Karma

thormanrd
Path Finder

We use size constraints on our hot/warm buckets and let the data tell us how long we can keep buckets accessible locally.  Once a size constraint is busted, the bucket will transition to cold which is smartstore on S3 for us.  That's the only place we apply a time settings for the final transition to frozen.  The time setting for transition to frozen (frozenTimePeriodInSecs) is set per index based on availability policy for that data.  For our analysts most indexes must be searchable for 90days, some longer.  When it transition happens, the S3 bucket is moved to Glacier storage on S3 which has to be thawed (a real PITA process) to make it searchable again.  Thawing data starts with finding it for a given index and timeframe, then moving it back to local storage and rebuilding the bucket (at least the metadata).

Tags (3)
0 Karma

panpanbebe
New Member

So what's the reason for cold tier storage if some customers do not use that at all? Because Archive storage will need to be thawed in order to bring to Hot/Warm or cold? I think maybe instead to bring to Hot/Warm, that's the use case for Cold Storage?

0 Karma

thormanrd
Path Finder

Cold buckets are still searchable.  The cold phase allows the admins to move data that is less likely to be searched to cheaper (i.e. slower) storage devices.  This allow for management of storage cost vs accessibility.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The cold tier is for data that rarely appears in search results.  The idea is one can put cold data on slower, cheaper storage to save operating costs.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

There is no single ideal retention for cold data.  It depends on your requirements and the storage devices available.  Typically, cold data is stored on the slowest devices, however, some customers do not use cold at all - data goes from warm directly to frozen.

The more important consideration, IMO, is the overall retention of data no matter where it is stored.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!