Need your help in understanding the reason behind the below behavior.
The data in my Index A is getting rolled over to Frozen bucket irrespective of the frozenTimePeriodInSecs set to 365 days. Here is my index configuration in indexes.conf.
[A] homePath = volume:primary/A/db coldPath = volume:primary/A/colddb thawedPath = $SPLUNK_DB/A/thaweddb coldToFrozenDir = /data/splunk/Splunk_Frozen_Data/ABC_APP/A/frozen frozenTimePeriodInSecs = 31536000
I could only see data available (searchable) ONLY for last 30 days in the system. I verified the 'coldToFrozenDir' path and found the rolled over buckets.
I followed the below Splunk Answer to figure out the reason behind the roll over, so that I can go and fix the problem.
In the result, I didn't find data related to Index A. It had results of every other index which were configured to rollover at specified time interval.
Also the result from the below search query confirmed data moved from cold bucket to frozen bucket
index=_internal sourcetype=splunkd bucketmover freeze "*A*" INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/data/splunk/var/lib/splunk/A/colddb/db_1492192312_1491334289_19'
I want to know what triggered the data to move from cold to frozen, when I set the index configuration not to rollover until surpassing 365 days.
Also, here are my default setting in indexes.conf.
[default] # Default for each index. Can be overridden per index based upon the volume of data received by that index. # 300GB homePath.maxDataSizeMB = 300000 # 200GB coldPath.maxDataSizeMB = 200000 # VOLUME SETTINGS # In this example, the volume spec is not defined here, it lives within # the org_(indexer|search)_volume_indexes app, see those apps for more # detail. # Option1: One Volume for Hot and Cold [volume:primary] path = /data/splunk/var/lib/splunk # 500GB maxVolumeDataSizeMB = 500000 [volume:frozen] path = /data/splunk/Splunk_Frozen_Data maxVolumeDataSizeMB = 500000
Appreciate your help. Thanks.
After referring to the Splunk documentation I think maxVolumeDataSizeMB configuration is causing cold buckets to roll over to frozen.
# Option1: One Volume for Hot and Cold [volume:primary] path = /data/splunk/var/lib/splunk # 500GB maxVolumeDataSizeMB = 500000
I believe cumulative size of my indexes is going beyond 500 GB which is causing Splunk to move cold buckets to frozen.
Can anyone tell me if there is a way to confirm this.
you can check the size of your cold volume and how full your cold volume is using searches leveraging | dbinspect or other method.
you can also check the DMC: settings -> indexing -> indexes and volumes: deployment
if its a single indexer, you can check the file system with df command
hope it helps
Yes. I logged into DMC and checked the size of Indexes and Volumes for both the indexers in indexer cluster. They were slightly above 500 GB. We will be adding more disk space our indexers and bump up the maxVolumeDataSizeMB to 90% of new disk space.
Please keep in mind that the implicit maximum index size – maxTotalDataSizeMB is also 500,000 MBs.
Are you saying we also need increase the value of maxTotalDataSizeMB to 90% of disk volume?
To sum up the comment thread,
used DMC and discovered that Splunk rolls buckets to frozen since the size of data in the cold volume exceeds the
here is a detailed explanation of this configuration from:
maxVolumeDataSizeMB = <positive integer> * Optional, ignored for storageType=remote * If set, this attribute limits the total size of all databases that reside on this volume to the maximum size specified, in MB. Note that this it will act only on those indexes which reference this volume, not on the total size of the path set in the path attribute of this volume. * If the size is exceeded, Splunk will remove buckets with the oldest value of latest time (for a given bucket) across all indexes in the volume, until the volume is below the maximum size. This is the trim operation. Note that this can cause buckets to be chilled [moved to cold] directly from a hot DB, if those buckets happen to have the least value of latest-time (LT) across all indexes in the volume. * Highest legal value is 4294967295, lowest legal value is 1.