Getting Data In

What kind of compression is used between forwarders and indexers?

bobwalden
Explorer

Just curious to know the format (gzip?) of the compression, and also the compression factor that can be expected?

Thanks,
bw

1 Solution

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.

View solution in original post

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...