Getting Data In

Can data compression of indexed data be switched off?

bhalberstadt2
New Member

I would like to know if data compression can be switched off entirely for indexers when writing data to storage. I am also interested in what the CPU load improvement could be for switching compression off. Please let me know if it is possible to switch it off.

0 Karma

davidpaper
Contributor

Raw data compression on an index can not be turned off. It can, however, have the algorithm used to compress be changed from the default (gzip) to lz4 as part of the index config in indexes.conf since 6.3.0. Note that this is per bucket, so you can change an existing index from the default to lz4 and back again. lz4 does not compress data as well as gzip, but uncompress is faster.

masonmorales
Influencer

FWIW, I tested LZ4 vs GZIP on indexers that had 10 GB/s disk read and >1M IOPS. Search performance was identical between the two. I was also told by another Splunker that LZ4 was implemented as part of a hackathon and should not be used in a production environment.

0 Karma

masonmorales
Influencer

Also, if you'd like to vote on a new feature to disable data compression, visit: https://ideas.splunk.com/ideas/EID-I-67

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...