How to generate storage and license usage reporting in a distributed Splunk deployment?


I have a License Master configured with 10 salves (about 5 Indexers and 5 forwarders).
Indexer1 - testindex1, testindex2,testindex3
Indexer2 - testindex4, testindex1, testindex5
Indexer3 - testindex1, testindex2, testindex6
sourcetypes - st1 (testindex1, testindex2), st2 (testindex3, tesindex4)
I have two license pools "LicensePool1"and "LicensePool2" of 500 MB each

The report I want to generate should have the following:
1. Overall license consumption by each Index/Host Vs License pool
2. Storage consumed by each index
3. Predict the license usage for one year, based on the consumption ( for each index/sourcetype/source)

I have got usage by Index using the search below:

index=_internal source=*license_usage.log type=Usage  | rename idx AS index  | timechart span=1d eval(round(sum(b)/1024/1024/1024,2)) AS "Total GB Used" by index

I need help in getting the search that provides such views.


I have almost got it after multiple tries:

| tstats max(Introspection.Disk_Objects.Capacity) AS Capacity max(Introspection.Disk_Objects.Avail) AS Avail max(Introspection.Disk_Objects.Used) AS Used max(Introspection.Disk_Objects.UsedPct) AS UsedPct FROM datamodel=Introspection_Usage WHERE nodename=Introspection.Disk_Objects [inputlookup all_servers.csv | fields host] GROUPBY _time host | eventstats max(_time) AS latest by host | where _time=latest | rename AS Mount |table host Mount Capacity Avail Used UsedPct | sort - UsedPct

however this query is giving only for LM. could anyone help in getting this extended for all slaves along with their roles?


Just an update after looking at masonmorales suggestion:
I have already evaluated Utilization monitor (SUM), Splunk on Splunk (SoS), Fire brigade apps. Have looked around most of the apps in my reach.
Specifically what I am looking for is a dashboard report that shows:

Host(Indexer) Index Storagevolume consumed, storage volume available, % used

Note that one host can have multiple indexes
I am trying to get this on License Master.
Some clue I have been looking at is using Dbinspect, using data from index=_introspection

0 Karma

Splunk Employee
Splunk Employee

The license usage logs identify your indexers by their guid, stored in the "i" field, so you will need to translate this into a hostname. The easiest way to do this is by hitting the "licenser/slave" endpoint on your license master and exporting the data to be used as a lookup table on the instance you're searching your license data. The example below assumes you are logged into the GUI on your license master and using the rest search command.

| rest /services/licenser/slaves splunk_server=local | fields title label | rename title AS guid, label AS indexer

This will generate a table to map guid to server name. Export this data using the export function or output it to a csv using loutputlookup or outputcsv commands. The end goal is to move it to your search instance as a lookup table, which will be used to enrich your license usage data. Once it is there, a search to determine the usage by indexer will be trivial.

You can use the predict command for forecasting.

earliest=-30d@d latest=-0d@d index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/license_usage.log" type=Usage | fields b  | timechart span=1d sum(b) AS b | predict b future_timespan=365

I am no expert on the predict command, so I suggest reading the documentation for tuning purposes.


thanks masonmorales I did have a look at this. however it doesn't satisfy my needs.

0 Karma


Thanks dcarmack , This helped a lot ..
However, I could not get the storage volume calculator from License master.
basically, I want to calculate how much of storage is consumed by each index/host

0 Karma