https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248961#M9386 <P>I had just setup Splunk with indexer clustering (RF-3, SF-2) with no data and initially loaded 1TB of syslog file using oneshot. The "Index Detail: Deployment" page showed that the total index size is 1121GB whereas the total raw data size (uncompressed) as 1783GB and hence the Raw to Index Size Ratio at 1.59:1.</P> <P>My question is how is it possible for 1024GB (1TB) file to be treated as 1783GB?</P> Sun, 22 Jan 2017 09:21:55 GMT srajarat2 2017-01-22T09:21:55Z https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248961#M9386 <P>I had just setup Splunk with indexer clustering (RF-3, SF-2) with no data and initially loaded 1TB of syslog file using oneshot. The "Index Detail: Deployment" page showed that the total index size is 1121GB whereas the total raw data size (uncompressed) as 1783GB and hence the Raw to Index Size Ratio at 1.59:1.</P> <P>My question is how is it possible for 1024GB (1TB) file to be treated as 1783GB?</P> Sun, 22 Jan 2017 09:21:55 GMT https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248961#M9386 srajarat2 2017-01-22T09:21:55Z https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248962#M9387 <P>The index size doesn't only depends upon the uncompressed raw data size. The Splunk create a compressed raw data files, as well as, a set of index files to make it searchable. The index consists of both these type of files. The compression ratio of raw data files and size of index files depends upon various factor. For more information, see following documentation. (see 2nd link for example of how Splunk calculates space).</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/HowSplunkstoresindexes">http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/HowSplunkstoresindexes</A><BR /> <A href="http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Systemrequirements#Storage_requirement_examples">http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Systemrequirements#Storage_requirement_examples</A></P> Sun, 22 Jan 2017 20:52:34 GMT https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248962#M9387 somesoni2 2017-01-22T20:52:34Z https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248963#M9388 <P>Sorry, if I was not clear. I am not asking about the index size. I do understand the sizing calculations on rawdata (RF) and tsidx (SF) in a clustered indexer mode. My question is specifically on the page "Index Detail: Deployment" page which shows the following information under "Index Structure Overview" (in 6.5.1).</P> <PRE><CODE> 8 (Indexers) 1121GB (Total Index Size) 1783GB (Total Raw Data size (uncompressed)) 1.59:1 (Raw to Index Size Ratio) </CODE></PRE> <P>My question is specifically on how Splunk measures the "Total Raw Data size (uncompressed)" as I just ingested a 1024GB syslog file and I was hoping to see that as the total raw data size and not 1783GB.</P> Mon, 23 Jan 2017 02:59:44 GMT https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248963#M9388 srajarat2 2017-01-23T02:59:44Z https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248964#M9389 <P>You can see the screenshot here.</P> <P><IMG src="http://www.somu.us/hidden_sisv/" alt="alt text" /></P> Mon, 23 Jan 2017 03:25:59 GMT https://community.splunk.com/t5/Deployment-Architecture/How-does-Splunk-estimate-the-total-raw-data-size/m-p/248964#M9389 srajarat2 2017-01-23T03:25:59Z