topic Re: Forwarder Output Compression Ratio in Getting Data In

Forwarder Output Compression Ratio- what is the expected bandwidth saving of a compressed stream if i activate it?

Matthias_BY — Mon, 28 Mar 2022 15:01:48 GMT

Hello,

i can activate compression on the universal forwarder to the indexer. as i understand from the documentation and some answers entries the compression is different between ssl encryption and not.

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

my question: what is the expected bandwidth saving of a compressed stream if i activate it? (useClientSSLCompression and non encrypted?

is it 1:10?

br
matthias

Re: Forwarder Output Compression Ratio

Dimitri_McKay — Tue, 25 Jun 2013 10:09:53 GMT

That would be contingent upon the repetition and amount of empty space in the data to be compressed.

Re: Forwarder Output Compression Ratio

bwooden — Tue, 25 Jun 2013 14:49:29 GMT

Compression is achieved via zlib. Per Dimitri's answer, compression is variable based on content.

Additional information may be found here: http://splunk-base.splunk.com/answers/63384/what-kind-of-compression-is-used-between-forwarders-and-indexers

Re: Forwarder Output Compression Ratio

hexx — Tue, 25 Jun 2013 22:09:40 GMT

What @bwooden and @Dimitri McKay is all true. Also, here are some rough numbers for compression ratio based on internal testing.

Universal forwarder / uncooked data
- no compression 1:1
- "native" compression 1:8
- SSL compression 1:14
Regular forwarder / cooked data
- no compression 1:1
- "native" compression 1:2
- SSL compression 1:8

Re: Forwarder Output Compression Ratio

Matthias_BY — Fri, 28 Jun 2013 15:03:13 GMT

thanks a lot for all your valuable feedback. Hexx received the points with clear numbers - of course depending on the content within a log it may vary a little bit - but if it is going over wan it's definitely worth to enable compression and even better SSL compression. 😉

Re: Forwarder Output Compression Ratio

andygerber — Mon, 28 Mar 2022 00:44:55 GMT

This is an old post, but as far as I can see still accurate.

Testing from an HF (azure data) to an indexer cluster, turning on SSL resulted in a 1:8 compression for very short (~65Byte) events.

Re: Forwarder Output Compression Ratio

adobrzeniecki_s — Tue, 15 Nov 2022 14:13:27 GMT

Hey Hexx, is there a specific doc that shows these compression ratios? This information is fantastic! Would also like a link to the docs page if you have one. Thank you so much.