Search to show license usage at heavy forwarder level

[preotesoiu]

Hello,

I have a search similar with below which provide a total of 2868 GB usage for last 24 hrs.

index=_internal source=*license_usage.log type=Usage splunk_server=indexer_server* | stats sum(b) AS bytes by splunk_server | eval GB= round(bytes/1024/1024/1024,3) | fields splunk_server GB | rename splunk_server as host | sort -GB  | addtotals row=f col=t labelfield="Total Indexed GB"

Also, I have such search for calculating thruput for HF and it has a total of 75.4 GB for last 24 hrs.

index="_internal" source="*metrics.log" group=per_host_thruput series=heavy_forwarder_server* | stats sum(kb) AS kb by series | eval GB= round(kb/1024/1024,3) | fields series GB | rename series as HF | addtotals row=f col=t labelfield="Total thruput GB"

All data comes through HF, it is cooked and sent upstream to the indexers.
Data comes in two way:
1. via Syslog - HF layer monitor the files and sends data to indexers.
2. via universal forwarders from downstream endpoints

Any suggestion on why such difference between license usage and thruput at Heavy Forwarders level?

Gabriel

[martin_mueller]

Two things come to mind:

• metrics logs are not complete, especially for high-cardinality serieses. Check how many distinct series values you have in one set of per_host_thruput, I'd expect that to be significantly lower than the number of hosts you actually have sending data in those 30 seconds.
• you're only looking at HFs, how much are the UFs doing?

To get per-host license usage, have you tried using the monitoring console? Indexing -> License Usage -> 30 days -> split by host

[mattymo]

Hey Gabe! I'm with Martin. Check out the REST calls the MC makes. See if your HFs are represented there as slaves. metrics.log can't be used as a complete record, only for operational analysis. Also tcpin_connections in _internal might be another option