Splunk Search

How to check the universal forwarder's metrics.log to get instantaneous_kbps and average_kbps?

kteng2024
Path Finder

From Documentation:

To verify how often the forwarder is hitting this limit, check the forwarder's metrics.log. (Look for this on the forwarder because metrics.log is not forwarded by default on universal and light forwarders.)

cd $SPLUNK_HOME/var/log/splunk/metrics.log
grep "name=thruput" metrics.log

Example: The instantaneous_kbps and average_kbps are always under 256KBps.

11-19-2013 07:36:01.398 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=251.790673,instantaneous_eps=3.934229, average_kbps=110.691774, total_k_processed=101429722, kb=7808.000000, ev=122

But when i run the grep -i "name=thruput" metrics.log , i wouldn't get any result. So, can i please know whether there is any way to check the instantaneous_kbps and average_kbps ?

0 Karma

woodcock
Esteemed Legend

Try this:
|tstats avg(tcp_KBps) AS "KBps" avg(tcp_eps) AS "Eps"
WHERE index=_internal AND sourcetype="splunkd" AND connectionType IN("cooked", "cookedSSL")
AND hostname="*"
BY hostname _time [| makeresults count=2
| streamstats count
| eval _time = _time - (100 * count)
| timechart count
| rename _* AS INV_*
| eval search = "span=" . INV_span . "s" ]
| rename hostname AS host
| timechart limit=0 useother=f first(KBps) AS KBps first(Eps) AS Eps BY host

0 Karma

woodcock
Esteemed Legend

Why bother going to the forwarder, just do this:

index=_internal source="*metrics.log" component="Metrics" host=MyHost
0 Karma

GregZillgitt
Path Finder

Can you load metrics.log into vi and search there?

I just ssh'd into one of our forwarders and it worked for me...

$ grep -i "name=thruput" metrics.log
01-26-2017 15:16:23.841 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.202834, instantaneous_eps=0.257975, average_kbps=19.430535, total_k_processed=4248380.000000, kb=6.290039, ev=8.000000, load_average=2.010000
01-26-2017 15:16:54.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.219149, instantaneous_eps=0.613138, average_kbps=19.427809, total_k_processed=4248386.000000, kb=6.791016, ev=19.000000, load_average=1.610000
01-26-2017 15:17:25.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.209961, instantaneous_eps=0.258064, average_kbps=19.425083, total_k_processed=4248392.000000, kb=6.508789, ev=8.000000, load_average=1.500000
0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...