Splunk Search

Calculating Splunk data Compression Size


Hi ,

I looked the daily ingestion for an index i am seeing total data ingested in last 7 days to an index is 800 GB. When i am calculating the total raw data size its showing total raw data ingested 1626 GB and its compressed to 759 GB which is at 46%. I am not understanding if i ingested 800 GB in last 7 days how come the raw total size data came to 1626 GB ? Any inputs will be appreciated.

Query using for compression:

| dbinspect index=xyz
| fields state,id,rawSize,sizeOnDiskMB
| stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB
|eval diskTotalinGB=(diskTotalinMB/1024)
| eval rawTotalinGB=(rawTotal / 1024 / 1024 / 1024) | fields - rawTotal
| eval compression=tostring(round(diskTotalinGB / rawTotalinGB * 100, 2)) + "%"
| table rawTotalinGB, diskTotalinGB, compression

Result: rawTotalinGB diskTotalinGB compression
1626.19525605347 759.39445495605 46.70%

Query used to calculate daily ingestion :
index=_internal source="license_usage.log" type=Usage idx=xyz| eval yearmonthday=strftime(_time, "%Y-%m-%d") | eval yearmonth=strftime(_time, "%Y-%m-%d") | stats sum(eval(b/1024/1024/1024)) AS volume_b by idx yearmonthday yearmonth | chart sum(volume_b) over yearmonth by idx|addcoltotals.

Which gives me total 862 GB ingestion in last 7 days.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...