We have a centralized license server running on a host called sl55splunk1. The result of the following search doesn't match the license portal:
index=_internal source=*license_usage.log host=sl55splunk1* type=Usage | timechart span=1d sum(b) as total | eval total=total/1024/1024/1024
Search Total=1979GB
License Portal=2289GB
Difference: 309GB
Which value is correct? I would expect the two values to match up. I have checked dates from November 25 through Dec. 20, and I see a daily variation that ranges anywhere from 59GB to 433GB. The Splunk Enterprise version is 6.2.6.
Problem:
License violation alert and license_usage.log reportedly did not match
Problem Summary:
A license violation alert had been sent to a customer on a particular day (2/27), but the license summary page reportedly did not indicate that any license was exceeded. The customer did receive an alert mail message saying that splunk indexed over 300GB on the previous day (2/26), but the summary page reportedly showed nothing.
What the customer did to troubleshoot the issue:
Questions from the customer:
How to Troubleshoot:
Resolution:
Provided customer with explanation as to why customer was receiving license alerts on Feb. 27, which are illustrated below:
Two alerts with different amounts of usage were seen on Feb. 27 because both slaves reported their usage to the master and each slave naturally reported different degrees of usage. Both these reports can be seen in the license_usage_summary.log report at the end of the day. (THIS LOG IS GENERATED AT THE END OF THE DAY BASED ON DATA FROM THE PREVIOUS DAY)
There is no relation to the splunkd.log errors customer provided in the creation of this ticket and the 400GB license usage alerts customer received from the DMC. The DMC alert is what correctly pointed out the +400GB license usage which can be found in license_usage_summary.log.
Are the license events indexed locally on the license master, or are you forwarding them on to a pool of indexers? Also, by license portal, are you referring to the license dashboard ran on the license master?
Thank you very much for the thought process on the reply. All events are forwarded to a pool of indexers. The License Master SH has a distributed search into each of the indexers. It turns out that 1 of the 5 was disabled. My search on the license master now matches the license dashboard on the license master. I enabled the distributed search to the 5th indexer. My search now shows as follows:
Search Total=2452GB
License Portal=2289GB
Difference: 163GB+
I am now thinking why the difference the other way.