New Splunk 8.1.1 feature - Enhanced TSIDX compression is related to tsidxWritingLevel ?

[edoardo_vicendo]

Hi,

I have seen this new feature in Splunk 8.1.1

https://docs.splunk.com/Documentation/Splunk/8.1.1/ReleaseNotes/MeetSplunk

Enhanced TSIDX compression

Enhanced TSIDX compression for improved performance and up to 40% reduced storage.

I was just wondering if this feature is related to the tsidxWritingLevel as mentioned here:

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/Reducetsidxdiskusage#The_tsidx_writing_le...

Splunk Enterprise Version	The tsidxWritingLevel supported	The supported tsidx level for searching
7.2.x	1, 2	1, 2
7.3.x	1, 2, 3	1, 2, 3
8.0.x	1, 2, 3	1, 2, 3
8.1.x	1, 2, 3, 4	1, 2, 3, 4

that can be set-up in indexes.conf as follow:

tsidxWritingLevel = [1|2|3|4]
* Enables various performance and space-saving improvements for tsidx files.
* For deployments that do not have multi-site index clustering enabled,
    set this to the highest value possible for all your indexes.
  * For deployments that have multi-site index clustering, only set
    this to the highest level possible AFTER all your indexers in the
    cluster have been upgraded to the latest code level.
  * Do not configure indexers with different values for 'tsidxWritingLevel'
    as downlevel indexers cannot read tsidx files created from uplevel peers.
  * The higher settings take advantage of newer tsidx file formats for
    metrics and log events that decrease storage cost and increase performance
* Default: 1

Moreover, is there anyone having some (good) feedback after have enabled it?

Thanks a lot,
Edoardo

[edoardo_vicendo]

@s2_splunk , @richgalloway thank you both for your answers, good to see they added additional information on the Release Note page and that my thought were right 😉

[richgalloway]

It's not clear from the Release Notes, is it?  Do what I did.  Submit feedback on the documentation page to let the Docs team know more information is needed.

[s2_splunk · Accepted solution]

FYI: Looks like docs have been enhanced to link to the relevant topic.

Please carefully read the docs on limitations with backwards compatibility. Buckets created with, for example, tsidxWritingLevel=4 cannot be read by versions that don't support that level.

[msplunk33]

We had very bad experience after we set 

tsidxwritinglevel = 4. We use 8.1.1. Our indexers having huge memory spike and started to crash later find out that there was bug with 8.1.1 (SPL-197930)

https://docs.splunk.com/Documentation/Splunk/8.1.4/ReleaseNotes/Fixedissues