topic Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license? in Getting Data In

Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

ankithreddy777 — Mon, 12 Dec 2016 20:59:33 GMT

I learned that Splunk compresses the incoming data and creates some index files to point towards compressed raw data. How does Splunk license account for this? Does Splunk charge license for uncompressed incoming data or for the compressed raw data?

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

somesoni2 — Mon, 12 Dec 2016 21:35:08 GMT

Splunk license is calculated as the amount of data indexed (which includes creating compressed raw data and index files).

Update

This actually is not true. See the comment below from @mmodestino_splunk.

All that matters is how much raw data makes it into the indexing pipeline (after any filtering).


Licensing is not how much data is in your index, it is how much data did the indexer have to index.

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

ankithreddy777 — Mon, 12 Dec 2016 21:40:38 GMT

suppose if there is 100GB incoming uncompressed data . splunk compressed the data to 15GB and created index files of 35GB , i.e 50GB stored on disk. So license will be charged for 100GB or 50GB

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

somesoni2 — Mon, 12 Dec 2016 21:52:13 GMT

It'll be 50 GB, which is basically what Splunk is saving under it's indexes. Besides, there are situations where you filter the data before indexing (to save license volume by removing junk data) OR mask some data , hence incoming data size can't be considered by license volume.

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

mattymo — Tue, 13 Dec 2016 15:29:17 GMT

it will be 100 GB. There is no need to confuse things with how splunk compresses and stores the data.

If you send 100GB to the indexing pipeline, you are using 100GB of license.

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

mattymo — Tue, 13 Dec 2016 15:31:45 GMT

I think we are confusing things a bit here...

All that matters is how much raw data makes it into the indexing pipeline (after any filtering).

Licensing is not how much data is in your index, it is how much data did the indexer have to index.

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

beatus — Tue, 13 Dec 2016 15:38:46 GMT

Splunk will calculate license volume based on the uncompressed size of the data, regardless of how that data makes it to Splunk. Compression features are there for cost savings at a storage level and network link level.

See: http://blogs.splunk.com/2016/05/06/what-size-should-my-splunk-license-be/

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

ankithreddy777 — Tue, 13 Dec 2016 15:45:39 GMT

Thank you, I have a file thats being ingested to splunk index. Why I am getting different values for the two queries below to find the daily ingestion rate.

index=_internal source=*license_usage.log type="Usage" | eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | stats sum(b) as b by idx| eval GB=(((b/1024)/1024))/1000

index=_internal source="/opt/splunk/var/log/splunk/metrics.log" series=index_name  | eval MB=kb/1024 | search group="per_index_thruput" | timechart span=1d sum(MB) by series

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

beatus — Tue, 13 Dec 2016 15:52:42 GMT

See here: http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

Metrics.log is only a sampling. license_usage.log is the ground truth.

Re: Does Splunk count the incoming uncompressed data, or the compressed raw data against your license?

somesoni2 — Tue, 13 Dec 2016 16:26:04 GMT

You're right. I got confused with index size vs license size. The uncompressed data going to indexing pipeline will indeed be the license usage.