We have tested TSIDX reduction on one index and want to implement retention on the reduced buckets

[sathwikr076]

Hello All,

We are having some storage capacity issues and trying some different things to make some space for the ingestion. So we did TSIDX reduction on one index as a test to see how much percentage of space we get if we do that and now we would like to implement retention on those reduced buckets. will this be a regular process of retention or will there be any problem in doing retention on the reduced buckets. Please let me know.

Thanks.

[vliggio]

Take a look here for more info about TSIDX reduction: https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Reducetsidxdiskusage . The problem will be reduced search performance (possibly significant) if you force removal of the TSIDX files. You didn't mention how much storage or ingestion you're talking about, or anything about the type of searches you do. I wouldn't recommend doing it for long term, because the reduced search performance will also mean reduced ingestion performance (since the indexer will now be spending more disk cycles on searching instead of indexing). If the buckets aren't used anymore (ie, you know buckets older than a month are not used), it wouldn't be too big an issue, but if you have a lot of searches being done across all time, then your performance will suffer.

If you have to have more space for indexing, you really should consider expanding your storage if possible (or adding another indexer if in a cluster). Storage performance can also go down significantly (depending on your file system and OS) as you get to the upper limits of your file system size.

[sathwikr076]

Thanks for the quick response. Unfortunately we are not provided with new hardware right now. We have implemented TSIDX reduction on the data which we don't need. We just tested it before implementing retention on that data but now we have a doubt if we implement retention on the reduced buckets, is it going to be same process as retention for regular buckets or will that be different.

[vliggio]

Retention is separate from TSIDX reduction. If you set a retention policy via time, that's for the bucket itself (on a per-index basis or via a global setting, and it's set by the frozenTimePeriodInSecs setting in your indexes.conf).

To set the tsidx reduction, you enable it via the following two values:
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction =

As long as your timePeriodInSecBeforeTsidxReduction is less than your frozenTimePeriodInSecs, the reduction will delete the full TSIDX files after the TsidxReduction number of seconds, and will retain the raw data and the mini TSIDX files. When the buckets age to the frozenTime, then the data itself will be deleted. The data will remain searchable until that frozenTime period, but will just be slower to search.

[sathwikr076]

ok great. Thanks for the response. we will try to implement the retention.