- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can data compression of indexed data be switched off?
I would like to know if data compression can be switched off entirely for indexers when writing data to storage. I am also interested in what the CPU load improvement could be for switching compression off. Please let me know if it is possible to switch it off.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Raw data compression on an index can not be turned off. It can, however, have the algorithm used to compress be changed from the default (gzip) to lz4 as part of the index config in indexes.conf since 6.3.0. Note that this is per bucket, so you can change an existing index from the default to lz4 and back again. lz4 does not compress data as well as gzip, but uncompress is faster.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

FWIW, I tested LZ4 vs GZIP on indexers that had 10 GB/s disk read and >1M IOPS. Search performance was identical between the two. I was also told by another Splunker that LZ4 was implemented as part of a hackathon and should not be used in a production environment.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Also, if you'd like to vote on a new feature to disable data compression, visit: https://ideas.splunk.com/ideas/EID-I-67
