How can I see why our license usage went up?

[ichesla1111]

Hello!

Last week (12/8/2022) my license usage went through the roof, for one source type that used 24 GB. On the other hand, when looking at the sourcetype, there were no events pulled into Splunk that day (no events since 9/16). What is the cause of this issue?? How can I see why our license usage went up??

Also
-->events pulled in that day (12/8) were same number of events we get pulled in on an average day YET our license usage was at 24.

Thank you.

[richgalloway]

It's not possible to consume license with no events so you should view those results with suspicion.

Bytes should be converted to GB *after* they have been summed.  Otherwise, you may be adding numbers that have been rounded to zero.

index=_internal source=*license_usage.log 
| stats count as events, sum(b) as bytes by st
| eval GB = round(bytes/1024/1024/1024, 3)
| table st events GB

To find the cause of the error, we need to run more searches to try to isolate the problem.  This search should identify the host sending the excess events. (Change "foo" to the name of the index in which the secret sourcetype is stored.)

| tstats count where sourcetype=<<secret sourcetype>> index=foo by host

Once you've identified the problem host, contact the owner/admin of that system to find out what changed right before the sudden spike in ingest.

This query can help spot the problem source.

| tstats count where sourcetype=<<secret sourcetype>> index=foo host=bar by source

[ichesla1111]

Thank you for getting back to me! When trying the searches you shared, it will not let me see data before 12/10 even when the search time frame is set to Month-Date. I need to see the data for 12/08 so I set the search time from the beginning to the end of the day of 12/08, yet it returns no results found. How can I get the search to show me data from 12/08?? Thank you.

 

[richgalloway]

It looks like your internal index is configured to save less than the default 30 days of events.  If that's the case then the data most likely is gone for good.

If your company archives the data then you may be able to restore it.  You would need to identify the buckets containing the events for the day of interest, load them into the appropriate "thawed" directory and rebuild the _internal index.