Splunk Search

Calculating Splunk data Compression Size


Hi ,

I looked the daily ingestion for an index i am seeing total data ingested in last 7 days to an index is 800 GB. When i am calculating the total raw data size its showing total raw data ingested 1626 GB and its compressed to 759 GB which is at 46%. I am not understanding if i ingested 800 GB in last 7 days how come the raw total size data came to 1626 GB ? Any inputs will be appreciated.

Query using for compression:

| dbinspect index=xyz
| fields state,id,rawSize,sizeOnDiskMB
| stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB
|eval diskTotalinGB=(diskTotalinMB/1024)
| eval rawTotalinGB=(rawTotal / 1024 / 1024 / 1024) | fields - rawTotal
| eval compression=tostring(round(diskTotalinGB / rawTotalinGB * 100, 2)) + "%"
| table rawTotalinGB, diskTotalinGB, compression

Result: rawTotalinGB diskTotalinGB compression
1626.19525605347 759.39445495605 46.70%

Query used to calculate daily ingestion :
index=_internal source="license_usage.log" type=Usage idx=xyz| eval yearmonthday=strftime(_time, "%Y-%m-%d") | eval yearmonth=strftime(_time, "%Y-%m-%d") | stats sum(eval(b/1024/1024/1024)) AS volume_b by idx yearmonthday yearmonth | chart sum(volume_b) over yearmonth by idx|addcoltotals.

Which gives me total 862 GB ingestion in last 7 days.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...