i want to find out how much license is getting used by cloud servers on which forwarders are present

[ksarode]

we are having a requirement where we want to know how much licensing usage is being used by cloud servers on which splunk universal forwarders are present.

[tiagofbmm]

You are forwarding your internal data of Splunk Forwaders to your Indexer layer.

If you check your internal index for license_usage.log and get a stats sum of license by host, you'll get that info for all the forwarders.

To get only the cloud ones, I suggest you filter that by their name

[ksarode]

can you help me with what query to use, our cloud servers name starts from cld-*

[ksarode]

when i am searching with below query
index=_internal source=*usage.log source="/opt/splunk/splunk/var/log/splunk/license_usage.log"
it is showing up only one host i.e our license master

[tiagofbmm]

You have to configure all your non Indexer instances (including your License Master) to forward data to the Indexer Layer.

Then in a Search Head that searches your indexers, you can do this

index=_internal source=*license_usage.log type=Usage earliest=@d
 | eval MB = b/1024/1024
 | rename st AS sourcetype
 | timechart span=1d sum(MB) AS "Total MB used" by host

[ksarode]

actually , this is giving me the license usage of one day. I am looking for the license daily used by the particular group of servers or you can say i want daily disk usage for the particular group of servers(cloud servers))

[tiagofbmm]

If you want to filter for a particular group of servers just put it in the beginning of the search

index=_internal host=cld* source=*license_usage.log type=Usage earliest=@d
  | eval MB = b/1024/1024
  | rename st AS sourcetype
  | timechart span=1d sum(MB) AS "Total MB used" by host

Don't forget to upvote useful comments

[ksarode]

its showing no result found. I think its not working

[tiagofbmm]

Check if you have internal data from those cloud forwarders in your indexers. If you don't, then you have to search that in the place that data is being sent.
More check your hosts name syntax if it matches he filter I put in the first line

[ksarode]

yes we are having internal data but the entry for particular source is not coming

[tiagofbmm]

Sorry you don't you have the license_usage.log of your hosts?

 index=_internal host=cld* source=*license_usage.log

Does this query return something or not?

[ksarode]

nope it has not returned anything.

when i tried with host=uslv* servers it is showing up results but not with the cld* ones

[ksarode]

can you tell me the reason behind this that why its not showing up for the cloud servers particularly?

[tiagofbmm]

That's what we are trying to diagnose. And the reason for my last comment

[ksarode]

under outputs.conf in one of the app present on cloud server its having below contents-

[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 5MB

[tcpout:primary_indexers]
server = uslv-papp-spk02.amgen.com:9997, uslv-papp-spk03.amgen.com:9997
autoLB = true

where spk02/03 are our indexers

[tiagofbmm]

OK so now the big question is, are your indexers receiving data from the Forwarders. Are you sure the cloud forwarders have connectivity to your indexers?

Are you sure that Hostname can be resolved by the Cloud Forwarders?

[ksarode]

yes i am receiving the internal logs for all the cloud servers which means indexers receiving data from the Forwarders

[tiagofbmm]

OK this is getting interesting. And we are sure they have been forwarding non-internal data (consuming license) recently?

[ksarode]

yes because basic cpu ,memory and disk is getting monitored for these servers

[tiagofbmm]

I'm afraid the license_usage.log is only generated by the License Master, and the structure info they have is only considering the indexer where the license is counted for, like this:

INFO  LicenseUsage - type=Usage s=who st=who h="tiago-VirtualBox" o="" idx="os" i="22EB6922-A37D-4586-A037-DC0E2D3FBCAD" pool="auto_generated_pool_enterprise" b=162 poolsz=10737418241

So it seems that kind of data is not generated by Splunk logs, and so not searchable by forwarder, only by host that hosts data, source, sourectype, idx, etc

[FrankVl]

The h field in license_usage should allow you to get the data usage per forwarder. Tiago: your example perhaps comes from a single instance lab environment? In a distributed environment, the license_usage log does record the license_usage per host (so if a UF is on the source host, host name = forwarder name = 'h' field in license usage log).