Even when running the default License Usage Report (LURV) on my indexer cluster, the numbers are reporting almost twice what I'm actually using.
index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d | eval _time=_time - 43200 | bin _time span=1d | stats latest(b) AS b by slave, pool, _time | timechart span=1d sum(b) AS "volume" fixedrange=false | join type=outer _time [search index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d | eval _time=_time - 43200 | bin _time span=1d | stats latest(stacksz) AS "stack size" by _time] | fields - _timediff | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]
Even trying to validate per host it shows twice:
index=customindex host=customhost | eval length = length(_raw) | stats sum(length)
With the help of Support, we were able to figure this out. My License Master was set up as distributed search, so multiple indexers were returning duplicates. This threw the numbers off. We turned off distributed search and set up the License Master as a Search Member to the Index Cluster and it all worked out.
1. Removed servers= from distsearch.conf on the license master 2. Restart Splunk 3. Configure the license master as a search head for an index cluster - Enable the search head - Click Settings in the upper right corner of Splunk Web. - In the Distributed environment group, click Indexer clustering. - Select Enable clustering. - Select Search head node and click Next. - Enter the Master URI including its management port. For example: https://10.0.0.0:8089 - Security key. This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster nodes. Set the same value here that you previously set on the master node. 4. Click Enable search head node. - The message appears, "You must restart Splunk for the search node to become active. You can restart Splunk from Server Controls." 5. Click Go to Server Controls. This takes you to the Settings page where you can initiate the restart.
Now my reports are accurate!
Thank you so much for this solution. I have been going insane trying to figure out why this was happening.