topic Re: What is the compression ratio between the forwarders and indexers? in Getting Data In

What is the compression ratio between the forwarders and indexers?

kreng — Wed, 09 Aug 2017 17:44:44 GMT

I need the approximate compression ratio of the data forwarded to indexers.

Re: What is the compression ratio between the forwarders and indexers?

lfedak_splunk — Wed, 09 Aug 2017 18:30:27 GMT

Hey @kreng, I saw this similar post and thought it might help answer your question: https://answers.splunk.com/answers/63384/what-kind-of-compression-is-used-between-forwarders-and-indexers.html

Re: What is the compression ratio between the forwarders and indexers?

adonio — Wed, 09 Aug 2017 20:37:14 GMT

Hello there,
Splunk estimates an average of 50% compression:
read here in detail:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Systemrequirements
also the link above provided by @lfdedak is good
you can always check for yourself:
bring the data to splunk and then search the following:

 | dbinspect index=*
    | fields state,id,rawSize,sizeOnDiskMB,index
    | stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index
    | eval rawTotalinMB=(rawTotal / 1024 / 1024) | fields - rawTotal
    | eval compression=tostring(round(100 - diskTotalinMB / rawTotalinMB * 100, 2)) + "%"

will suggest to ignore indexes with very little data as this takes into calculation metadata files within the index and therefore you might see a huge negative compression on these tiny indexes.
hope it helps

Re: What is the compression ratio between the forwarders and indexers?

jkat54 — Wed, 09 Aug 2017 20:41:18 GMT

To add to the discussion here, For ssl compression we generally go with 13 to 1.