topic Re: Trying to understand compression - given % compression of X volume data, how much on disk is required? in Monitoring Splunk

Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine — Thu, 17 Oct 2013 21:03:31 GMT

I'm trying to understand the compression numbers provided by Splunk. Given a compression of, say, 40%, on a volume of 100 GB. What does that translate to on disk for storage purposes?

Is it 60 GB (100 GB x (100-40)) OR
is it 40% x 100 GB = 40 GB?
Or, something else?

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

_d_ — Thu, 17 Oct 2013 21:17:36 GMT

Not clear what number's you're referring to but compression ratio should always be calculated as:

Compression Ratio = (Uncompressed Size)/(Compressed Size)

Also, compression ratio expressed in percent does not make much sense. Storage savings on the other hand are a different story.

Example:

Uncompressed = 100GB, Compressed = 40GB

compression ratio = 100/40 = 2.5 OR alternatively noted as 2.5:1

savings % = 100 * (1 - 40GB/100GB) = 60%

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine — Thu, 17 Oct 2013 22:55:49 GMT

I checked the stats and the numbers work out to your answer.

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine — Thu, 17 Oct 2013 23:10:51 GMT

diskTotalinMB = rawTotalinMB * (compression * 100)

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

sowings — Fri, 18 Oct 2013 00:06:12 GMT

Compression expressed as percent absolutely makes sense! "The data takes up X% of its original volume." 100GB indexed at 40% compression rate = 40GB on disk.

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

_d_ — Fri, 18 Oct 2013 01:46:21 GMT

No. You're confusing the compressed size of something expressed as a percentage of the original with compression ratio of a certain mechanism. Mathematically and technically speaking, the compression ratio is never expressed or noted in percent. Instead notations such as x/y, x:y or alike are used.
"100GB indexed at 40% compression rate = 40GB on disk." This is a wrongly worded statement about percentages and rates. That's like saying: a $100 pair of jeans sold at 25% discount will cost $25.

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

_d_ — Fri, 18 Oct 2013 01:49:42 GMT

...char limit...A more exact statement would be your former one: "100GB of raw data indexed takes up 40% of its original volume"

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine — Fri, 18 Oct 2013 18:39:48 GMT

Also, is this a reliable number to use for storage calculation? What I mean is, does this diskTotalinMB include all associated files that require space for that index? Is this the "du" for the entire index (hot and cold) and all files within?

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

the_wolverine — Mon, 17 Feb 2014 03:33:22 GMT

That's why this is confusing -- the wording is wrong.

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

jlaw — Wed, 08 Oct 2014 19:42:32 GMT

This docs topic talks about calculating storage:

http://docs.splunk.com/Documentation/Splunk/latest/Installation/Estimateyourstoragerequirements

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

hakeniz — Fri, 18 Sep 2020 01:19:36 GMT

Splunk Document Link Changed. Follow below Link for your reference :

https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Estimateyourstoragerequirements

Re: Trying to understand compression - given % compression of X volume data, how much on disk is required?

edoardo_vicendo — Wed, 08 Jan 2025 15:15:32 GMT

See my reply here if it can help

https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013