Getting Data In

What kind of compression is used between forwarders and indexers?

bobwalden
Explorer

Just curious to know the format (gzip?) of the compression, and also the compression factor that can be expected?

Thanks,
bw

1 Solution

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.

View solution in original post

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...