Archive

whats happens if "maxVolumeDataSizeMB" limit is reached for cold path.

Contributor

Hi,

if "maxVolumeDataSizeMB" is reached for home volume , the buckets roll to cold.

what is "maxVolumeDataSizeMB" is reached for the cold path. Does it causes cold buckets to frozen or does it stops indexing.

If it is frozen, Even though we set frozentimeperiodinsecs to high value. It effect is no more?

0 Karma

SplunkTrust
SplunkTrust

From indexes.conf

maxVolumeDataSizeMB = <positive integer>
.......
*) ** If the size is exceeded, Splunk will remove buckets with the oldest value
  of latest time (for a given bucket) across all indexes in the volume,
  until the volume is below the maximum size.  This is the trim operation.
  Note that this can cause buckets to be chilled [moved to cold] directly
  from a hot DB, if those buckets happen to have the least value of
  latest-time (LT) across all indexes in the volume.***

Contributor

If data is frozen, Even though we set frozentimeperiodinsecs to high value. Its effect is no more if "maxVolumeDataSizeMB" is reached? So only way to save the data is by configuring coldtofrozendir.

0 Karma

SplunkTrust
SplunkTrust

Are you using index clustering? If not clustered then coldtofrozendir is easiest way. If clustered it is a lot more complex and likely involves a professional services engagement.

0 Karma

Contributor

we are using the indexer clustering. It is okay even though we store multiple copies of frozen data . Can we go for coldtofrozenDir.

0 Karma

SplunkTrust
SplunkTrust

If you use that in a cluster you should make it shared storage all the indexers can see that has a LOT of room. Use a sub folder per indexer. You will get duplicate buckets this way. That is why it's complex. You then have to come up with some programatic way to dedup the frozen buckets that get dumped from multiple indexers to reduce your storage consumption after they land.

0 Karma

SplunkTrust
SplunkTrust

Splunk should never stop indexing. It will however throw stuff to frozen and if you don't take measures to ensure what frozen means or where it goes that means gone.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/HowSplunkstoresindexes

Contributor

If it is frozen, Even though we set frozentimeperiodinsecs to high value. It effect is no more if "maxVolumeDataSizeMB" is reached?

0 Karma

SplunkTrust
SplunkTrust

The data retention policy is enforced using two parameters, the age (frozenTimePeriodInSecs) and size (maxTotalDataSizeMB or maxVolumeDataSizeMB) whichever happens first.

(https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setaretirementandarchivingpolicy#Set_attr...

0 Karma