Getting Data In

What is the compression ratio between the forwarders and indexers?

kreng
New Member

I need the approximate compression ratio of the data forwarded to indexers.

0 Karma

adonio
Ultra Champion

Hello there,
Splunk estimates an average of 50% compression:
read here in detail:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Systemrequirements
also the link above provided by @lfdedak is good
you can always check for yourself:
bring the data to splunk and then search the following:

 | dbinspect index=*
    | fields state,id,rawSize,sizeOnDiskMB,index
    | stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index
    | eval rawTotalinMB=(rawTotal / 1024 / 1024) | fields - rawTotal
    | eval compression=tostring(round(100 - diskTotalinMB / rawTotalinMB * 100, 2)) + "%"

will suggest to ignore indexes with very little data as this takes into calculation metadata files within the index and therefore you might see a huge negative compression on these tiny indexes.
hope it helps

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Hey @kreng, I saw this similar post and thought it might help answer your question: https://answers.splunk.com/answers/63384/what-kind-of-compression-is-used-between-forwarders-and-ind...

0 Karma

jkat54
SplunkTrust
SplunkTrust

To add to the discussion here, For ssl compression we generally go with 13 to 1.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...