Archive
Highlighted

Events are not moving from WARM to COLD bucket

Explorer

Hi Team,

Am testing index archiving and disk space allocation with below settings in indexes.conf file. Problem here is I don’t see any data moving into cold bucket, from warm bucket but its directly archived & forzen.
Please let me know is there any issue with the configuration.

indexes.conf

[mytest]
homePath   = $SPLUNK_DB\mytest\db
coldPath   = $SPLUNK_DB\mytest\colddb
thawedPath = $SPLUNK_DB\mytest\thaweddb
tstatsHomePath = volume:_splunk_summaries\mytest\datamodel_summary

# The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
maxDataSize = 1

# The maximum size of an index (in MB). This parameter only applies to hot, warm, and cold buckets.  It does not apply to thawed buckets.
maxTotalDataSizeMB = 7

# Specifies the maximum size of homePath (which contains hot and warm buckets).
homePath.maxDataSizeMB = 2

# If this size is exceeded, Splunk will freeze buckets with the oldest value of latest time (for a given bucket) 
#     until coldPath is below the maximum size. 
coldPath.maxDataSizeMB = 5

# Controls the service period (in seconds): how often splunkd performs certain housekeeping tasks.
rotatePeriodInSecs = 60

# path to frozen archive
coldToFrozenDir = C:\Users\Downloads\SplunkForzen

# Number of seconds after which indexed data rolls to frozen. 4 hours
frozenTimePeriodInSecs = 14400

Thanks
Mugunthan

Tags (1)
Highlighted

Re: Events are not moving from WARM to COLD bucket

Ultra Champion

Well, you run with very small sizes here, And I think that the hot+warm buckets may exceed the limit for the index in total. As it says in the docs, these limits are approximate to allow for splunk to make its own optimizations etc.

Try increasing the figures by a factor of 10 for all the size-related parameters.

Also, you are aware that it is the first of either maxTotalDataSizeMB and frozenTimePeriodInSecs that matches, that will trigger a bucket move to frozen, so perhaps you want to increase the time limit as well (to ensure that your moves are being triggered by the size parameters).

/k

0 Karma
Highlighted

Re: Events are not moving from WARM to COLD bucket

Explorer

Hi Team,

Please confirm does below seeting are correct and will work. My home path size is 3 gb (1 gb hot and 2 gb Warm), cold path size is 4gb and total size of the index is 7gb. Planned to purge 7 days old data.

[mytest]
homePath   = $SPLUNK_DB\mytest\db
coldPath   = $SPLUNK_DB\mytest\colddb
thawedPath = $SPLUNK_DB\mytest\thaweddb
tstatsHomePath = volume:_splunk_summaries\mytest\datamodel_summary
# The maximum size of an index (in MB). This parameter only applies to hot, warm, and cold buckets.  It does not apply to thawed buckets.
maxTotalDataSizeMB = 7168

# The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
maxDataSize = 1024

# Specifies the maximum size of homePath (which contains hot and warm buckets).
homePath.maxDataSizeMB = 3072

# If this size is exceeded, Splunk will freeze buckets with the oldest value of latest time (for a given bucket) 
#     until coldPath is below the maximum size. 
coldPath.maxDataSizeMB = 4096

# Number of seconds after which indexed data rolls to frozen. 7 days
frozenTimePeriodInSecs = 604800

Thanks
Mugunthan

0 Karma
Highlighted

Re: Events are not moving from WARM to COLD bucket

Ultra Champion

Well, these sizes are still not very large, but I do think that they should work. Given the size of each bucket (1 GB) there will still be times when you are 'over the limit' with up to a GB or so (or even more, depending on the number of simultaneous hot buckets).

I don't think you need to make settings for all three *DataSizeMB's. Just two of them would make third implicitly set.

Just go with these settings and see where it takes you, but I really don't see why you keep it so small. Are you running this off a 486?

0 Karma
Highlighted

Re: Events are not moving from WARM to COLD bucket

Explorer

What was your outcome? I just did something similar and what I experienced was the cold bucket data doubled and in some cases tripled or more in size. This was due to the fact that I changed maxWarmDBCount to 299, but why?

0 Karma
Highlighted

Re: Events are not moving from WARM to COLD bucket

Explorer

Thanks Kristian.

We are having 8 indexes and need to share 179Gb disk space, so we have limited the disk space for each index. Also we are running with Rehat Linux 6.0.

Thanks
Mugunthan

0 Karma