I recently introduced a few parameters around different buckets like hot, warm, cold, etc.
Now I need to see if the buckets are rotating based on the values I provided, and I am trying to find an effective search to help.
Parameters I recently introduced and want to validate based on bucket size and movement are:
maxDataSize = auto_high_volume maxHotBuckets = 10 maxWarmDBCount = 15 maxTotalDataSizeMB = 512000 frozenTimePeriodInSecs = 7776000
The maxTotalDataSizeMB parameter does not apply to individual buckets. It is the maximum data size that an entire index can consume. Once this is reached, the oldest data will roll to frozen. Since the default behavior for rolling to frozen is deletion, you can potentially lose data.
To be sure you are aware, if you change the bucket settings on an existing index the following may be true/likely:
That means if you had 6 years of data, each year with 10,000 buckets, and you change the settings to be 100 buckets per year, Splunk will create 100 buckets per year going forward in time. It will not change the OLD buckets.
However if you change the retention TIME to 1 year on the same data, Splunk WILL delete the older buckets (but not always immediately). Last bit is key, it can take time to delete buckets in a large environment. Your search might produce unexpected results until all the fix up tasks were completed.
Finally, you'll have to allow time for a good sample set of new buckets to be created with the new settings before you can find the settings are correct/incorrect using the command already mentioned:
| dbinspect index=yourIndex
Using your talent, developing the requirements and then use dbinspect and other SPL commands to achieve your requirements.
"Monitor these changes since the day I change them..." Is too vague for anyone to just give you a solution with expectations of it being what you're looking for.
"I want to know if buckets grow larger than maxTotalDataSizeMB"... Now that's a requirement someone can help with.
I have several queries but not helping much.
I need to determine with help of some effective search query on how buckets state changed and their counts in individual states after I changed these parameters -
maxDataSize = auto_high_volume
maxHotBuckets = 10
maxWarmDBCount = 15
maxTotalDataSizeMB = 512000
frozenTimePeriodInSecs = 7776000