Installation

Why doesn't the license_usage.log summed value match the license usage portal?

Path Finder

We have a centralized license server running on a host called sl55splunk1. The result of the following search doesn't match the license portal:

index=_internal source=*license_usage.log host=sl55splunk1* type=Usage | timechart span=1d sum(b) as total | eval total=total/1024/1024/1024 

Search Total=1979GB
License Portal=2289GB
Difference: 309GB

Which value is correct? I would expect the two values to match up. I have checked dates from November 25 through Dec. 20, and I see a daily variation that ranges anywhere from 59GB to 433GB. The Splunk Enterprise version is 6.2.6.

Labels (1)
0 Karma

Splunk Employee
Splunk Employee

Problem:

License violation alert and license_usage.log reportedly did not match

Problem Summary:

A license violation alert had been sent to a customer on a particular day (2/27), but the license summary page reportedly did not indicate that any license was exceeded. The customer did receive an alert mail message saying that splunk indexed over 300GB on the previous day (2/26), but the summary page reportedly showed nothing.

What the customer did to troubleshoot the issue:

  • Checked indexed license_usage.log in relation to Feb 27 from SH, but most of indexed data was already deleted due to misconfiguration.
  • Configurations were fixed by PS and the customer can search entire indexed data since March 7th.

Questions from the customer:

  • How did the license master calculate the amount of data indexed on Feb 27th
  • Did this unexpected license violation is occur due to the general error messages we found in splunkd.log.

How to Troubleshoot:

  • navigate to license_usage_summary.log'
  • grep -rin --color "RolloverSummary" (or the date) "02-27" license_usage_summary.log
  • look for the number behind the 'b' which is the amount of data actually indexed on the previous day.

Resolution:

Provided customer with explanation as to why customer was receiving license alerts on Feb. 27, which are illustrated below:

  • Two alerts with different amounts of usage were seen on Feb. 27 because both slaves reported their usage to the master and each slave naturally reported different degrees of usage. Both these reports can be seen in the license_usage_summary.log report at the end of the day. (THIS LOG IS GENERATED AT THE END OF THE DAY BASED ON DATA FROM THE PREVIOUS DAY)

  • There is no relation to the splunkd.log errors customer provided in the creation of this ticket and the 400GB license usage alerts customer received from the DMC. The DMC alert is what correctly pointed out the +400GB license usage which can be found in license_usage_summary.log.

0 Karma

Splunk Employee
Splunk Employee

Are the license events indexed locally on the license master, or are you forwarding them on to a pool of indexers? Also, by license portal, are you referring to the license dashboard ran on the license master?

0 Karma

Path Finder

Thank you very much for the thought process on the reply. All events are forwarded to a pool of indexers. The License Master SH has a distributed search into each of the indexers. It turns out that 1 of the 5 was disabled. My search on the license master now matches the license dashboard on the license master. I enabled the distributed search to the 5th indexer. My search now shows as follows:

Search Total=2452GB
License Portal=2289GB
Difference: 163GB+

I am now thinking why the difference the other way.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!