How to generate storage and license usage reporting in a distributed Splunk deployment?

[bkumarm]

I have a License Master configured with 10 salves (about 5 Indexers and 5 forwarders).
Indexer1 - testindex1, testindex2,testindex3
Indexer2 - testindex4, testindex1, testindex5
Indexer3 - testindex1, testindex2, testindex6
sourcetypes - st1 (testindex1, testindex2), st2 (testindex3, tesindex4)
I have two license pools "LicensePool1"and "LicensePool2" of 500 MB each

The report I want to generate should have the following:
1. Overall license consumption by each Index/Host Vs License pool
2. Storage consumed by each index
3. Predict the license usage for one year, based on the consumption ( for each index/sourcetype/source)

I have got usage by Index using the search below:

index=_internal source=*license_usage.log type=Usage  | rename idx AS index  | timechart span=1d eval(round(sum(b)/1024/1024/1024,2)) AS "Total GB Used" by index

I need help in getting the search that provides such views.

[bkumarm]

I have almost got it after multiple tries:

| tstats max(Introspection.Disk_Objects.Capacity) AS Capacity max(Introspection.Disk_Objects.Avail) AS Avail max(Introspection.Disk_Objects.Used) AS Used max(Introspection.Disk_Objects.UsedPct) AS UsedPct FROM datamodel=Introspection_Usage WHERE nodename=Introspection.Disk_Objects [inputlookup all_servers.csv | fields host] GROUPBY _time host Introspection.data.mount_point | eventstats max(_time) AS latest by host | where _time=latest | rename Introspection.data.mount_point AS Mount |table host Mount Capacity Avail Used UsedPct | sort - UsedPct

however this query is giving only for LM. could anyone help in getting this extended for all slaves along with their roles?

[bkumarm]

Just an update after looking at masonmorales suggestion:
I have already evaluated Utilization monitor (SUM), Splunk on Splunk (SoS), Fire brigade apps. Have looked around most of the apps in my reach.
Specifically what I am looking for is a dashboard report that shows:

Host(Indexer) Index Storagevolume consumed, storage volume available, % used

Note that one host can have multiple indexes
I am trying to get this on License Master.
Some clue I have been looking at is using Dbinspect, using data from index=_introspection

[dcarmack_splunk]

The license usage logs identify your indexers by their guid, stored in the "i" field, so you will need to translate this into a hostname. The easiest way to do this is by hitting the "licenser/slave" endpoint on your license master and exporting the data to be used as a lookup table on the instance you're searching your license data. The example below assumes you are logged into the GUI on your license master and using the rest search command.

| rest /services/licenser/slaves splunk_server=local | fields title label | rename title AS guid, label AS indexer

This will generate a table to map guid to server name. Export this data using the export function or output it to a csv using loutputlookup or outputcsv commands. The end goal is to move it to your search instance as a lookup table, which will be used to enrich your license usage data. Once it is there, a search to determine the usage by indexer will be trivial.

You can use the predict command for forecasting.

earliest=-30d@d latest=-0d@d index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/license_usage.log" type=Usage | fields b  | timechart span=1d sum(b) AS b | predict b future_timespan=365

I am no expert on the predict command, so I suggest reading the documentation for tuning purposes.

http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Predict

[bkumarm]

thanks masonmorales I did have a look at this. however it doesn't satisfy my needs.

[bkumarm]

Thanks dcarmack , This helped a lot ..
However, I could not get the storage volume calculator from License master.
basically, I want to calculate how much of storage is consumed by each index/host

[masonmorales]

https://splunkbase.splunk.com/app/2678/