- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: Index buckets configuration using time
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, dear ninjas!
I need to configure my indexes to store data in bucket using time periods.
For example:
Index - Test
Hot/warm buckets have to store data for 60 days then move it to cold buckets
Cold buckets should store data for 120 days (+60 from warm buckets) = 180 days then move outdated data to Frozen
Frozen have to store it 180 days (+ 180 days from cold buckets) and after 360 days delete the outdated data.
I didn't find options in default indexes.conf for that. Also should I write a script which will move data from cold to frozen? Doesn't Splunk do it automatically?
Reference
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
frozen. (https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf)
Thank you!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You are looking at the situation completely wrong. You should buy as much fast SSD as you can afford for hot space. Then, depending on sizing estimates of your data (https://splunk-sizing.appspot.com/) acquire slower storage to use for older cold, ensuring that you have enough to meet your retention goals. If you are OK with deleting data after that, then you do not need frozen space at all; frozen buckets are not searchable and are intended to be backups for emergencies/audits because there is an arduous thawing process to make them searchable again.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You are looking at the situation completely wrong. You should buy as much fast SSD as you can afford for hot space. Then, depending on sizing estimates of your data (https://splunk-sizing.appspot.com/) acquire slower storage to use for older cold, ensuring that you have enough to meet your retention goals. If you are OK with deleting data after that, then you do not need frozen space at all; frozen buckets are not searchable and are intended to be backups for emergencies/audits because there is an arduous thawing process to make them searchable again.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, but how do I specify storing in days?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think you are missing something here ...
frozen data is not searchable and not being handled by splunk anymore.
you can control the retention of your frozen data by calculating its daily growth and your disk / storage size.
as for the other configurations, use indexs.conf attributes and values to setup according to requirements.
check this to create your relevant configurations for time and size retention:
https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Indexesconf
note, there are other important variables like: how much data you ingest every day and, if you have a cluster, what is the replication and search factors. you will have to pay close attention to those when configuring your index
Monitoring MariaDB and MySQL
Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...
Splunk Federated Analytics for Amazon Security Lake
