topic Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index in Getting Data In

Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

karthikklv — Fri, 19 May 2017 17:24:33 GMT

Hi All,

Need your help in understanding the reason behind the below behavior.
The data in my Index A is getting rolled over to Frozen bucket irrespective of the frozenTimePeriodInSecs set to 365 days. Here is my index configuration in indexes.conf.

[A]
homePath = volume:primary/A/db
coldPath = volume:primary/A/colddb
thawedPath = $SPLUNK_DB/A/thaweddb
coldToFrozenDir = /data/splunk/Splunk_Frozen_Data/ABC_APP/A/frozen
frozenTimePeriodInSecs = 31536000

I could only see data available (searchable) ONLY for last 30 days in the system. I verified the 'coldToFrozenDir' path and found the rolled over buckets.

I followed the below Splunk Answer to figure out the reason behind the roll over, so that I can go and fix the problem.
[https://answers.splunk.com/answers/117988/halp-my-data-is-being-rolled-to-frozen-and-i-dont-know-why.html][1]

In the result, I didn't find data related to Index A. It had results of every other index which were configured to rollover at specified time interval.

Also the result from the below search query confirmed data moved from cold bucket to frozen bucket

index=_internal sourcetype=splunkd bucketmover freeze "*A*"
INFO  BucketMover - AsyncFreezer freeze succeeded for bkt='/data/splunk/var/lib/splunk/A/colddb/db_1492192312_1491334289_19'

I want to know what triggered the data to move from cold to frozen, when I set the index configuration not to rollover until surpassing 365 days.

Also, here are my default setting in indexes.conf.

[default]
# Default for each index. Can be overridden per index based upon the volume of data received by that index.
# 300GB
homePath.maxDataSizeMB = 300000
# 200GB
coldPath.maxDataSizeMB = 200000

# VOLUME SETTINGS
# In this example, the volume spec is not defined here, it lives within
# the org_(indexer|search)_volume_indexes app, see those apps for more
# detail.

# Option1: One Volume for Hot and Cold
[volume:primary]
path = /data/splunk/var/lib/splunk
# 500GB
maxVolumeDataSizeMB = 500000

[volume:frozen]
path = /data/splunk/Splunk_Frozen_Data
maxVolumeDataSizeMB = 500000

Appreciate your help. Thanks.

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

karthikklv — Fri, 19 May 2017 18:40:54 GMT

After referring to the Splunk documentation I think maxVolumeDataSizeMB configuration is causing cold buckets to roll over to frozen.

 # Option1: One Volume for Hot and Cold
 [volume:primary]
 path = /data/splunk/var/lib/splunk
 # 500GB
 maxVolumeDataSizeMB = 500000

I believe cumulative size of my indexes is going beyond 500 GB which is causing Splunk to move cold buckets to frozen.

Can anyone tell me if there is a way to confirm this.

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

adonio — Sat, 20 May 2017 22:09:08 GMT

yes,
you can check the size of your cold volume and how full your cold volume is using searches leveraging | dbinspect or other method.
you can also check the DMC: settings -> indexing -> indexes and volumes: deployment
if its a single indexer, you can check the file system with df command
hope it helps

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

ddrillic — Sun, 21 May 2017 00:50:03 GMT

Please keep in mind that the implicit maximum index size – maxTotalDataSizeMB is also 500,000 MBs.

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

karthikklv — Sun, 21 May 2017 16:26:56 GMT

Yes. I logged into DMC and checked the size of Indexes and Volumes for both the indexers in indexer cluster. They were slightly above 500 GB. We will be adding more disk space our indexers and bump up the maxVolumeDataSizeMB to 90% of new disk space.

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

karthikklv — Sun, 21 May 2017 16:36:31 GMT

Are you saying we also need increase the value of maxTotalDataSizeMB to 90% of disk volume?

Re: Data getting rollover to Frozen bucket irrespective of frozenTimePeriodInSecs set to 365 days (31536000 secs) for the index

adonio — Mon, 22 May 2017 00:47:16 GMT

To sum up the comment thread,
used DMC and discovered that Splunk rolls buckets to frozen since the size of data in the cold volume exceeds the maxVolumeDataSizeMB
here is a detailed explanation of this configuration from:
https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Indexesconf

maxVolumeDataSizeMB = <positive integer>
* Optional, ignored for storageType=remote
* If set, this attribute limits the total size of all databases that reside
  on this volume to the maximum size specified, in MB.  Note that this it
  will act only on those indexes which reference this volume, not on the
  total size of the path set in the path attribute of this volume.
* If the size is exceeded, Splunk will remove buckets with the oldest value
  of latest time (for a given bucket) across all indexes in the volume,
  until the volume is below the maximum size.  This is the trim operation.
  Note that this can cause buckets to be chilled [moved to cold] directly
  from a hot DB, if those buckets happen to have the least value of
  latest-time (LT) across all indexes in the volume.
* Highest legal value is 4294967295, lowest legal value is 1.