Getting Data In

What is the compression ratio between the forwarders and indexers?

New Member

I need the approximate compression ratio of the data forwarded to indexers.

0 Karma

Ultra Champion

Hello there,
Splunk estimates an average of 50% compression:
read here in detail:
also the link above provided by @lfdedak is good
you can always check for yourself:
bring the data to splunk and then search the following:

 | dbinspect index=*
    | fields state,id,rawSize,sizeOnDiskMB,index
    | stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index
    | eval rawTotalinMB=(rawTotal / 1024 / 1024) | fields - rawTotal
    | eval compression=tostring(round(100 - diskTotalinMB / rawTotalinMB * 100, 2)) + "%"

will suggest to ignore indexes with very little data as this takes into calculation metadata files within the index and therefore you might see a huge negative compression on these tiny indexes.
hope it helps

0 Karma

Splunk Employee
Splunk Employee

Hey @kreng, I saw this similar post and thought it might help answer your question:

0 Karma


To add to the discussion here, For ssl compression we generally go with 13 to 1.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!