Are you a member of the Splunk Community?
- Find Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- Re: What happens when maxVolumeDataSizeMB paramet...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Our maxVolumeDataSizeMB parameter for the warm-bucketss is set to 1.9 TB. What happens if I set this parameters to 1.5 TB? Will the warm-buckets moved to cold after the restart?
Greetz,
Marc
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The non hot buckets containing the oldest events will be frozen to meet the requirement.
Will the warm-buckets moved to cold after the restart
NO, they will be frozen (i.e. deleted if no frozen location or script is defined) not moved from warm to cold.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I can say from recent experience in 2017 on Splunk 6.5 that the buckets will roll from warm to cold, even if the cold volume and warm volume are the same.
One of my customer had their warm maxVolumeDataSizeMB set to 1.5TB, and their cold maxVolumeDataSizeMB also set to 1.5TB. However, they only had 1.5TB of disk space available, TOTAL!
We reduced the maxVolumeDataSizeMB for both hot and cold to 725GB each. Upon deploying the bundle from the cluster master, through the monitoring console, we can instantly see the warm bucket size and usage decrease to 725GB, and the size of the cold bucket increase to 725GB. We also validated by looking at the filesystem on one of the indexers.
The old data was NOT frozen!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The non hot buckets containing the oldest events will be frozen to meet the requirement.
Will the warm-buckets moved to cold after the restart
NO, they will be frozen (i.e. deleted if no frozen location or script is defined) not moved from warm to cold.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I can say from recent experience in 2017 on Splunk 6.5 that the buckets will roll from warm to cold, even if the cold volume and warm volume are the same.
A customer maximized their disk space, and the indexer stopped writing to disk. They reduced the maxVolumeDataSizeMB setting on the cluster master, and redeployed the bundle to the indexers. Through the monitoring console, we can instantly see the warm bucket size and usage decrease, and the size of the cold bucket increase usage increase. We also validated by looking at the filesystem.
The old data was NOT frozen!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This seems to contradict you.
When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
oops. 2nd part of the dirname, but the 1st epoch. my bad. corrected above already.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You need to define a frozen path, so that splunk knows where to put them (and not delete them). However, once they are frozen, they are no longer searchable, as only the raw data, and not the tsidx files are retained.
You can figure out which of the buckets that are likely to be frozen.
This will require that you find the size reduction / bucket size number of buckets and move them manually. In your case that could be (400 GB / 10 GB) ~40 buckets, but you should probably take a few more. Take the 50 buckets with the lowest values of X, where X is the 1st epoch timestamp part of the dirname
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can I find out wich buckets are frozen and move them to cold?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Community Content Calendar, September edition
Splunkbase Unveils New App Listing Management Public Preview
Leveraging Automated Threat Analysis Across the Splunk Ecosystem
