I have a search similar with below which provide a total of 2868 GB usage for last 24 hrs.
index=_internal source=*license_usage.log type=Usage splunk_server=indexer_server* | stats sum(b) AS bytes by splunk_server | eval GB= round(bytes/1024/1024/1024,3) | fields splunk_server GB | rename splunk_server as host | sort -GB | addtotals row=f col=t labelfield="Total Indexed GB"
Also, I have such search for calculating thruput for HF and it has a total of 75.4 GB for last 24 hrs.
index="_internal" source="*metrics.log" group=per_host_thruput series=heavy_forwarder_server* | stats sum(kb) AS kb by series | eval GB= round(kb/1024/1024,3) | fields series GB | rename series as HF | addtotals row=f col=t labelfield="Total thruput GB"
All data comes through HF, it is cooked and sent upstream to the indexers.
Data comes in two way:
1. via Syslog - HF layer monitor the files and sends data to indexers.
2. via universal forwarders from downstream endpoints
Any suggestion on why such difference between license usage and thruput at Heavy Forwarders level?
Two things come to mind:
To get per-host license usage, have you tried using the monitoring console? Indexing -> License Usage -> 30 days -> split by host
Hey Gabe! I'm with Martin. Check out the REST calls the MC makes. See if your HFs are represented there as slaves. metrics.log can't be used as a complete record, only for operational analysis. Also
_internal might be another option