Getting Data In

Can data compression of indexed data be switched off?

bhalberstadt2
New Member

I would like to know if data compression can be switched off entirely for indexers when writing data to storage. I am also interested in what the CPU load improvement could be for switching compression off. Please let me know if it is possible to switch it off.

0 Karma

davidpaper
Contributor

Raw data compression on an index can not be turned off. It can, however, have the algorithm used to compress be changed from the default (gzip) to lz4 as part of the index config in indexes.conf since 6.3.0. Note that this is per bucket, so you can change an existing index from the default to lz4 and back again. lz4 does not compress data as well as gzip, but uncompress is faster.

masonmorales
Influencer

FWIW, I tested LZ4 vs GZIP on indexers that had 10 GB/s disk read and >1M IOPS. Search performance was identical between the two. I was also told by another Splunker that LZ4 was implemented as part of a hackathon and should not be used in a production environment.

0 Karma

masonmorales
Influencer

Also, if you'd like to vote on a new feature to disable data compression, visit: https://ideas.splunk.com/ideas/EID-I-67

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...