What kind of compression is used between forwarders and indexers?

[bobwalden]

Just curious to know the format (gzip?) of the compression, and also the compression factor that can be expected?

Thanks,
bw

[hexx · Accepted solution]

This mostly depends on one fact: Are you forwarding using SSL or not?

• If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

• If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.