Getting Data In

Can data compression of indexed data be switched off?

bhalberstadt2
New Member

I would like to know if data compression can be switched off entirely for indexers when writing data to storage. I am also interested in what the CPU load improvement could be for switching compression off. Please let me know if it is possible to switch it off.

0 Karma

davidpaper
Contributor

Raw data compression on an index can not be turned off. It can, however, have the algorithm used to compress be changed from the default (gzip) to lz4 as part of the index config in indexes.conf since 6.3.0. Note that this is per bucket, so you can change an existing index from the default to lz4 and back again. lz4 does not compress data as well as gzip, but uncompress is faster.

masonmorales
Influencer

FWIW, I tested LZ4 vs GZIP on indexers that had 10 GB/s disk read and >1M IOPS. Search performance was identical between the two. I was also told by another Splunker that LZ4 was implemented as part of a hackathon and should not be used in a production environment.

0 Karma

masonmorales
Influencer

Also, if you'd like to vote on a new feature to disable data compression, visit: https://ideas.splunk.com/ideas/EID-I-67

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...