Which value is correct? I would expect the two values to match up. I have checked dates from November 25 through Dec. 20, and I see a daily variation that ranges anywhere from 59GB to 433GB. The Splunk Enterprise version is 6.2.6.
License violation alert and license_usage.log reportedly did not match
A license violation alert had been sent to a customer on a particular day (2/27), but the license summary page reportedly did not indicate that any license was exceeded. The customer did receive an alert mail message saying that splunk indexed over 300GB on the previous day (2/26), but the summary page reportedly showed nothing.
What the customer did to troubleshoot the issue:
Checked indexed license_usage.log in relation to Feb 27 from SH, but most of indexed data was already deleted due to misconfiguration.
Configurations were fixed by PS and the customer can search entire indexed data since March 7th.
Questions from the customer:
How did the license master calculate the amount of data indexed on Feb 27th
Did this unexpected license violation is occur due to the general error messages we found in splunkd.log.
How to Troubleshoot:
navigate to license_usage_summary.log'
grep -rin --color "RolloverSummary" (or the date) "02-27" license_usage_summary.log
look for the number behind the 'b' which is the amount of data actually indexed on the previous day.
Provided customer with explanation as to why customer was receiving license alerts on Feb. 27, which are illustrated below:
Two alerts with different amounts of usage were seen on Feb. 27 because both slaves reported their usage to the master and each slave naturally reported different degrees of usage. Both these reports can be seen in the license_usage_summary.log report at the end of the day. (THIS LOG IS GENERATED AT THE END OF THE DAY BASED ON DATA FROM THE PREVIOUS DAY)
There is no relation to the splunkd.log errors customer provided in the creation of this ticket and the 400GB license usage alerts customer received from the DMC. The DMC alert is what correctly pointed out the +400GB license usage which can be found in license_usage_summary.log.
Are the license events indexed locally on the license master, or are you forwarding them on to a pool of indexers? Also, by license portal, are you referring to the license dashboard ran on the license master?
Thank you very much for the thought process on the reply. All events are forwarded to a pool of indexers. The License Master SH has a distributed search into each of the indexers. It turns out that 1 of the 5 was disabled. My search on the license master now matches the license dashboard on the license master. I enabled the distributed search to the 5th indexer. My search now shows as follows: