Deployment Architecture

We have tested TSIDX reduction on one index and want to implement retention on the reduced buckets

sathwikr076
Communicator

Hello All,

We are having some storage capacity issues and trying some different things to make some space for the ingestion. So we did TSIDX reduction on one index as a test to see how much percentage of space we get if we do that and now we would like to implement retention on those reduced buckets. will this be a regular process of retention or will there be any problem in doing retention on the reduced buckets. Please let me know.

Thanks.

Labels (2)
0 Karma

vliggio
Communicator

Take a look here for more info about TSIDX reduction: https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Reducetsidxdiskusage . The problem will be reduced search performance (possibly significant) if you force removal of the TSIDX files. You didn't mention how much storage or ingestion you're talking about, or anything about the type of searches you do. I wouldn't recommend doing it for long term, because the reduced search performance will also mean reduced ingestion performance (since the indexer will now be spending more disk cycles on searching instead of indexing). If the buckets aren't used anymore (ie, you know buckets older than a month are not used), it wouldn't be too big an issue, but if you have a lot of searches being done across all time, then your performance will suffer.

If you have to have more space for indexing, you really should consider expanding your storage if possible (or adding another indexer if in a cluster). Storage performance can also go down significantly (depending on your file system and OS) as you get to the upper limits of your file system size.

sathwikr076
Communicator

Thanks for the quick response. Unfortunately we are not provided with new hardware right now. We have implemented TSIDX reduction on the data which we don't need. We just tested it before implementing retention on that data but now we have a doubt if we implement retention on the reduced buckets, is it going to be same process as retention for regular buckets or will that be different.

0 Karma

vliggio
Communicator

Retention is separate from TSIDX reduction. If you set a retention policy via time, that's for the bucket itself (on a per-index basis or via a global setting, and it's set by the frozenTimePeriodInSecs setting in your indexes.conf).

To set the tsidx reduction, you enable it via the following two values:
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction =

As long as your timePeriodInSecBeforeTsidxReduction is less than your frozenTimePeriodInSecs, the reduction will delete the full TSIDX files after the TsidxReduction number of seconds, and will retain the raw data and the mini TSIDX files. When the buckets age to the frozenTime, then the data itself will be deleted. The data will remain searchable until that frozenTime period, but will just be slower to search.

sathwikr076
Communicator

ok great. Thanks for the response. we will try to implement the retention.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...