Splunk App for Stream: How to find the total amount of data being indexed from forwarders for a 24 hour period?

[helius]

Hola,

To avoid filling up our license, I'd like to find the total amount of data being indexed from the forwarders for a 24 hour period. I'm new to my environment, splunk, and was not sure where to start. Anyone got any suggestions?

[mdickey_splunk · Accepted solution]

This may not be as accurate as the other methods, but as an alternative you can also just run a search on the raw events to get a feel for index usage:

source=stream* | eval l=len(_raw) | stats sum(l) as BytesSum | eval TotalMB=round(BytesSum/(1024*1024), 0) | fields - BytesSum

(for 1 day, use something like "Yesterday" or "Last 24 hours" for the search time range)

[helius]

Sweet, exactly what I was looking for. I'll study the string and see what you did. Appreciate it!

[mdickey_splunk]

Search adapted from http://answers.splunk.com/answers/168412/how-to-determine-index-volume-by-sourcetype.html:

 earliest=-1d@d latest=@d  index=_internal source=*license_usage.log* type=Usage st=stream*
 | stats sum(b) AS Bytes by st 
 | sort -Bytes

I just added "st=stream*" to only include data generated by stream.

[helius]

Thanks. However, there is no type=Usage in any of the license_usage.logs in my index on any of the indexers or even the search head.

[jsie_splunk]

This will depend on what you've actually configured in terms of enabled Streams, enabled fields, etc.

Have you tried looking at the License Usage Report? The "Previous 30 days" tab plus "Split By Source/Source type" may offer the insight you need.

[helius]

Hmm, odd. Our Monitor App doesn't have anything.

Manually running the search string (Inspect) doesn't work either. I wonder if we've configured this properly...

[jsie_splunk]

I was using the License Usage report here: http://yoursplunkserver:8000/en-US/manager/search/licenseusage

You can access via Settings -> System -> Licensing.

And then the green Usage Report button.

[helius]

I'm now seeing "[subsearch]: No matching fields exist" at the top of splunk when I go to the License Usage page.