topic Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space in Splunk Search

compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

jan_wohlers — Thu, 30 Aug 2012 08:27:35 GMT

Hey,

we just set up a indexer 3 weeks ago. By now we are indexing about 50gig/24h. If I go to Manager -> Indexes I can see that our main index only has a size of about 100gigs. Mostly just eventlogs will be indexed. Is there such a good compression that about 20 days of 50gigs/day will be stored in an 100gig index?

Thanks for your answer in advance!

Jan

Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

MuS — Thu, 30 Aug 2012 08:47:22 GMT

Hi jan.wohlers

basically you can say compression between 40-50% are normal, you can check this with this search:

| dbinspect index=_internal
| fields state,id,rawSize,sizeOnDiskMB 
| stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB
| eval rawTotalinMB=(rawTotal / 1024 / 1024) | fields - rawTotal
| eval compression=tostring(round(diskTotalinMB / rawTotalinMB * 100, 2)) + "%"
| table rawTotalinMB, diskTotalinMB, compression

cheers,

MuS

Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

jan_wohlers — Thu, 30 Aug 2012 08:57:40 GMT

Okay, thanks for the answer. Compressionrate is 21% which seems pretty good.

Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

ibondarets — Tue, 12 May 2015 08:24:08 GMT

Thank you for this handy search example!
I've couple of questions regarding it:
1) how can i build a search which will give me a table of all indexes present wih compression ratio information? I tried this:

| dbinspect index=*
  | fields state,id,rawSize,sizeOnDiskMB 
  | stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index
  | eval rawTotalinMB=(rawTotal / 1024 / 1024) | fields - rawTotal
5.  | eval compression=tostring(round(diskTotalinMB / rawTotalinMB * 100, 2)) + "%"
  | table rawTotalinMB, diskTotalinMB, compression

but it didn't work.
2) what does it mean when I get this:

Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

ConnorG — Tue, 12 May 2015 20:50:18 GMT

I have a similar compression percentage as ibondarets.

Guess that means our data is actually larger once indexed.

Re: compression rate of indexed data: 50gig/day in 3 weeks uses 100gig HDD space

edoardo_vicendo — Wed, 08 Jan 2025 15:16:00 GMT

See my reply here if it can help

https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013