Why does my Splunk License usage report have blank values for some days?

[splunker12er]

I have a Splunk license usage report scheduled on a monthly basis, (usage by host on a daily basis)

Usage details per host:

index=_internal source=*license_usage.log* type=Usage 
| eval s=if(s=="","unknown",s)
| eval h=if(h=="","unknown",h)
| timechart span=1d sum(b) AS volume_b by h

Great. Here, I am able to get the Splunk License usage consumption per host on a daily basis, but I do see for some days, the values are blank (neither '0'). Does this mean:

1. The hosts haven't connected to the license master ?
2. The hosts haven't sent any logs to the indexer for indexing ?

Your kind reply is much helpful.
Thanks.

[dvg06]

I was trying to forward license_usage.log from license master server but that was not working.
Had to move this log file to some other directory, and then it started getting indexed.

cp etc/log.cfg etc/log-local.cfg

vim etc/log-local.cfg
line 247, updated to: appender.license_usage.fileName=${SPLUNK_HOME}/var/log/license_usage.log
Restarted splunk, and it started working...

[yannK]

This is totally possible,
- The host haven't connected to the license master ?
- The host haven't sent any logs to the indexer for indexing ?
If an license-slave is unable to connect to the license-master, it will send all the report in at the same time when it can reconnect.

Another possibility is that the logs in "source=license_usage.log"
- have not been indexed for some reason (rotating too quickly, or wrong permissions , or forwarding issue)
- or have been indexed, then were deleted from the _internal index for retention reasons (time or size)

[splunker12er]

Thanks for the reply.

Actually , I do see the trend map of the license_usage.log for more than 2 months. (retention policy has been extended)

Time range : April to Jun

I see missing values/blank in between the range.

[yannK]

This is expected, the _internal index has only 30 days retention by default (and 500GB maximum size per indexer),
if you see any data older than one month, this is just because some buckets have still not yet been deleted because they overlap the retention.

conclusion : no you cannot check old data in the index.
If you need to keep longer, you can change your retention for next time, or setup some type of summary indexing/acceleration to store in another index.

workaround : maybe do you still have the physical log files on disk (splunk keeps 5 rotated files up to 25MB each, see in $SPLUNK_HOME/var/log/splunk/license_usage.*)

[splunker12er]

Is this actually due to the hosts haven't forward any data for indexing ? So license_usage.log file doesn't contain any events during the time-range ?

[splunker12er]

just for info., we don't see any connectivity failure between splunk and hosts