Getting Data In

Index buckets configuration using time

damiko
Communicator

Hello, dear ninjas!
I need to configure my indexes to store data in bucket using time periods.
For example:
Index - Test
Hot/warm buckets have to store data for 60 days then move it to cold buckets
Cold buckets should store data for 120 days (+60 from warm buckets) = 180 days then move outdated data to Frozen
Frozen have to store it 180 days (+ 180 days from cold buckets) and after 360 days delete the outdated data.

I didn't find options in default indexes.conf for that. Also should I write a script which will move data from cold to frozen? Doesn't Splunk do it automatically?
Reference
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
frozen. (https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf)
Thank you!

0 Karma
1 Solution

woodcock
Esteemed Legend

You are looking at the situation completely wrong. You should buy as much fast SSD as you can afford for hot space. Then, depending on sizing estimates of your data (https://splunk-sizing.appspot.com/) acquire slower storage to use for older cold, ensuring that you have enough to meet your retention goals. If you are OK with deleting data after that, then you do not need frozen space at all; frozen buckets are not searchable and are intended to be backups for emergencies/audits because there is an arduous thawing process to make them searchable again.

View solution in original post

woodcock
Esteemed Legend

You are looking at the situation completely wrong. You should buy as much fast SSD as you can afford for hot space. Then, depending on sizing estimates of your data (https://splunk-sizing.appspot.com/) acquire slower storage to use for older cold, ensuring that you have enough to meet your retention goals. If you are OK with deleting data after that, then you do not need frozen space at all; frozen buckets are not searchable and are intended to be backups for emergencies/audits because there is an arduous thawing process to make them searchable again.

damiko
Communicator

Yes, but how do I specify storing in days?

0 Karma

adonio
Ultra Champion

I think you are missing something here ...
frozen data is not searchable and not being handled by splunk anymore.
you can control the retention of your frozen data by calculating its daily growth and your disk / storage size.
as for the other configurations, use indexs.conf attributes and values to setup according to requirements.
check this to create your relevant configurations for time and size retention:
https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf

note, there are other important variables like: how much data you ingest every day and, if you have a cluster, what is the replication and search factors. you will have to pay close attention to those when configuring your index

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...