Seems my license master was down over the holiday period, not really a big deal as it's mostly for testing atm. However when I started it back up I immediately got a license alert telling me we had indexed 180GB in the last day. My license is currently 100GB.
If I use the following search: index=_internal source=*metrics.log group=per_index_thruput series!=_* | eval totalGB = (kb/1024)/1024 | timechart span=1d sum(totalGB)
looking at a few of the Splunk apps like SOS it also shows my usage at 180GB under license usage, although looking at what has actually been indexed using status > indexer activity > indexer activity overview or indexing and forwarding view in SOS it shows me similar results to the above search.
So it seems to be some kind of mismatch between the ways the licensing is being seen by splunk, Anyone have any suggestion on what may be going on here or how I would go about getting to the bottom of this?
The information used to determine license violations is calculated on a "RolloverSummary" type. This is updated nightly at midnight. Other calculations can be made using a type of "Usage" which occurs continuously. If, for various reasons, the "RolloverSummary" information is not received by the license server, it will be queued up for the next successful connection. In cases like these multiple days worth of "RolloverSummary" data will submitted for a single 24 hour period causing license violation warnings to be displayed.
It is important to make sure communication between indexers and License Masters is not impeded around midnight to prevent these types of occurrences.