https://community.splunk.com/t5/Getting-Data-In/What-kind-of-compression-is-used-between-forwarders-and-indexers/m-p/103240#M21693 <P>This mostly depends on one fact: Are you forwarding using SSL or not?</P> <UL> <LI><STRONG>If you are not using SSL</STRONG> to forward, there is no compression enabled by default, so you'll have to set <CODE>compressed = true</CODE> in the appropriate stanzas of the indexer's <A href="http://docs.splunk.com/Documentation/Splunk/5.0/admin/inputsconf">inputs.conf</A>:</LI> </UL> <P><CODE>compressed = [true|false]<BR /> * Specifies whether receiving compressed data.<BR /> * If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.<BR /> * Defaults to false.</CODE></P> <P>... and in the forwarder's <A href="http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Outputsconf">outputs.conf</A>:</P> <P><CODE>compressed = [true|false]<BR /> * Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.<BR /> * If true, forwarder sends compressed data.<BR /> * If set to true, the receiver port must also have compression turned on (in its inputs.conf file).<BR /> * Defaults to false.</CODE></P> <P>In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.</P> <UL> <LI><STRONG>If you are using SSL</STRONG> to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be <STRONG>a lot</STRONG> higher than for "native" compression because zlib will operate on much larger chunks of data.</LI> </UL> Fri, 26 Oct 2012 00:38:18 GMT hexx 2012-10-26T00:38:18Z https://community.splunk.com/t5/Getting-Data-In/What-kind-of-compression-is-used-between-forwarders-and-indexers/m-p/103239#M21692 <P>Just curious to know the format (gzip?) of the compression, and also the compression factor that can be expected? </P> <P>Thanks,<BR /> bw</P> Thu, 25 Oct 2012 21:58:03 GMT https://community.splunk.com/t5/Getting-Data-In/What-kind-of-compression-is-used-between-forwarders-and-indexers/m-p/103239#M21692 bobwalden 2012-10-25T21:58:03Z https://community.splunk.com/t5/Getting-Data-In/What-kind-of-compression-is-used-between-forwarders-and-indexers/m-p/103240#M21693 <P>This mostly depends on one fact: Are you forwarding using SSL or not?</P> <UL> <LI><STRONG>If you are not using SSL</STRONG> to forward, there is no compression enabled by default, so you'll have to set <CODE>compressed = true</CODE> in the appropriate stanzas of the indexer's <A href="http://docs.splunk.com/Documentation/Splunk/5.0/admin/inputsconf">inputs.conf</A>:</LI> </UL> <P><CODE>compressed = [true|false]<BR /> * Specifies whether receiving compressed data.<BR /> * If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.<BR /> * Defaults to false.</CODE></P> <P>... and in the forwarder's <A href="http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Outputsconf">outputs.conf</A>:</P> <P><CODE>compressed = [true|false]<BR /> * Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.<BR /> * If true, forwarder sends compressed data.<BR /> * If set to true, the receiver port must also have compression turned on (in its inputs.conf file).<BR /> * Defaults to false.</CODE></P> <P>In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.</P> <UL> <LI><STRONG>If you are using SSL</STRONG> to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be <STRONG>a lot</STRONG> higher than for "native" compression because zlib will operate on much larger chunks of data.</LI> </UL> Fri, 26 Oct 2012 00:38:18 GMT https://community.splunk.com/t5/Getting-Data-In/What-kind-of-compression-is-used-between-forwarders-and-indexers/m-p/103240#M21693 hexx 2012-10-26T00:38:18Z