License usage stats in monitoring console seem very off

[Robbie1194]

Hi guys,

So I've noticed that when we go into the monitoring console and view the license usage over the previous 30 days, it works fine as is. However, if I change it to split by index/sourcetype/etc, the figures change drastically and are no where near correct.

For example, say our daily license is 300gb, it says that ONE of our indexes used 570gb that day, not to mention our other 8 or so indexes.

We have a search head cluster that can run this search:

index=_internal source=*license_usage.log type="Usage"
| eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h)
| eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s)
| eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)
| bin _time span=1d
| stats sum(b) as b by _time, pool, s, st, h, idx
| timechart span=1d sum(b) AS volumeB by idx fixedrange=false
| join type=outer _time [search index=_internal source=*license_usage.log type="RolloverSummary" earliest=-30d@d
| eval _time=_time - 43200
| bin _time span=1d
| stats latest(stacksz) AS "stack size" by _time]
| fields - _timediff
| foreach * [eval <>=round('<>'/1024/1024/1024, 3)]
| fields - "stack size"
| addtotals

and get the correct figures for license usage by index/sourcetype/etc. But the monitoring console's figures don't match when using this search.

I think our license master sends it's internal logs to our indexers so I don't understand why the mc can't query it but the shc can? Anyone got any ideas? I'm not too clued up on how all the license usage stuff works so if anyone has a better understanding, some explanations would be appreciated!

Cheers!

[mdsnmss]

I have actually noticed issues with that myself and created an app for license monitoring because of it (https://splunkbase.splunk.com/app/3576/). I reverse engineered the Monitoring Console's searches and that base component of the search is like the one you posted. From what I've seen in the Monitoring Console, it appears to double each value when in it split. This makes it seem that the data is getting returned twice. It sounds like your setup is similar to mine where the internal logs get sent to the indexers. My guess is running the search on the License Master accesses the internal logs locally, as well as from the indexers and returns the same data twice.