topic Index buckets configuration using time in Getting Data In

Index buckets configuration using time

damiko — Fri, 25 Oct 2019 08:31:02 GMT

Hello, dear ninjas!
I need to configure my indexes to store data in bucket using time periods.
For example:
Index - Test
Hot/warm buckets have to store data for 60 days then move it to cold buckets
Cold buckets should store data for 120 days (+60 from warm buckets) = 180 days then move outdated data to Frozen
Frozen have to store it 180 days (+ 180 days from cold buckets) and after 360 days delete the outdated data.

I didn't find options in default indexes.conf for that. Also should I write a script which will move data from cold to frozen? Doesn't Splunk do it automatically?
Reference
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
frozen. (https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf)
Thank you!

Re: Index buckets configuration using time

adonio — Sat, 26 Oct 2019 16:29:40 GMT

I think you are missing something here ...
frozen data is not searchable and not being handled by splunk anymore.
you can control the retention of your frozen data by calculating its daily growth and your disk / storage size.
as for the other configurations, use indexs.conf attributes and values to setup according to requirements.
check this to create your relevant configurations for time and size retention:
https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf

note, there are other important variables like: how much data you ingest every day and, if you have a cluster, what is the replication and search factors. you will have to pay close attention to those when configuring your index

Re: Index buckets configuration using time

woodcock — Sat, 26 Oct 2019 21:53:31 GMT

You are looking at the situation completely wrong. You should buy as much fast SSD as you can afford for hot space. Then, depending on sizing estimates of your data (https://splunk-sizing.appspot.com/) acquire slower storage to use for older cold, ensuring that you have enough to meet your retention goals. If you are OK with deleting data after that, then you do not need frozen space at all; frozen buckets are not searchable and are intended to be backups for emergencies/audits because there is an arduous thawing process to make them searchable again.

Re: Index buckets configuration using time

damiko — Wed, 30 Oct 2019 06:26:43 GMT

Yes, but how do I specify storing in days?