Splunk License Usage: Splunk license master report less volume than usage search

[Motoko89]

Hi all, I have a non-clustered distributed deployment: 1 head, 3 indexers, 1 deployment + license master + monitoring. I have already configured the Monitoring Console to Distributed mode. However I notice that the license usage under Settings > Licensing and the License Usage in Monitoring Console show only around a third of the data I get from the below query:

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool

I also notice that in Licensing, under Indexers, only 1 indexer is shown at any time. The indexers are connected to the License master fine. I checked and they have been checking in frequently.

Any idea what I missed here? Thanks,

[saurabh_tek11]

@Motoko89, If the LM is showing less value of indexed data today and it differs with your search query, apparently 3 indexers are not part of same license pool OR one/more indexers are running out-of-the-box license.

Check the connected indexers with this LM on your LM machine at Settings > Licensing » Connected indexers.

--
1. Have you connected all three indexers (license slaves) with the remote License master to share the same license pool?

You can check this in your 3 indexers under Settings > Licensing. If done recently, you would have to restart splunkd.

1. Is it the same machine, sharing DS, LM and MC ?
   Since MC is essentially a SH overlooking all Splunk instances in your infrastructure, you would need to "add search peers to the search head" i.e. you would need to add 4 peers on this machine - 1 SH and 3 indexer machine to start reading their internal logs.
   Hope you have done that as you mentioned MC is in distributed mode.
   Still, please check in Settings > Distributed search » Search peers > SH and 3 indexers

2. Are you forwarding _internal logs from SH and MC to all 3 indexers individually OR as you designed (since you are not in clustered environment) ?

It seems your MC is reading _internal log only from one indexer and not from all. So there is limited amount of data showing limited results.

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool

Provide me the number of hosts and splunk_server in your above search query.

[saurabh_tek11]

Hi @Motoko89, Did you check it.

[Motoko89]

1. They have been indexing 100s of GB of data for 8 months now. I checked
2. Let me check this
3. Yes, _internal logs from search head is forwarded

[MousumiChowdhur]

Hi @Motoko89,

Can you please confirm, if you have non- clustered environment, how do you get data in your system for indexing? I'm assuming that you are seeing just one third of the license usage because you might be getting data in through only one of your indexers.

[Motoko89]

Through universal forwarders. They have been sending data to all 3 for 8 months now