Splunk Search

How to check the universal forwarder's metrics.log to get instantaneous_kbps and average_kbps?

kteng2024
Path Finder

From Documentation:

To verify how often the forwarder is hitting this limit, check the forwarder's metrics.log. (Look for this on the forwarder because metrics.log is not forwarded by default on universal and light forwarders.)

cd $SPLUNK_HOME/var/log/splunk/metrics.log
grep "name=thruput" metrics.log

Example: The instantaneous_kbps and average_kbps are always under 256KBps.

11-19-2013 07:36:01.398 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=251.790673,instantaneous_eps=3.934229, average_kbps=110.691774, total_k_processed=101429722, kb=7808.000000, ev=122

But when i run the grep -i "name=thruput" metrics.log , i wouldn't get any result. So, can i please know whether there is any way to check the instantaneous_kbps and average_kbps ?

0 Karma

woodcock
Esteemed Legend

Try this:
|tstats avg(tcp_KBps) AS "KBps" avg(tcp_eps) AS "Eps"
WHERE index=_internal AND sourcetype="splunkd" AND connectionType IN("cooked", "cookedSSL")
AND hostname="*"
BY hostname _time [| makeresults count=2
| streamstats count
| eval _time = _time - (100 * count)
| timechart count
| rename _* AS INV_*
| eval search = "span=" . INV_span . "s" ]
| rename hostname AS host
| timechart limit=0 useother=f first(KBps) AS KBps first(Eps) AS Eps BY host

0 Karma

woodcock
Esteemed Legend

Why bother going to the forwarder, just do this:

index=_internal source="*metrics.log" component="Metrics" host=MyHost
0 Karma

GregZillgitt
Path Finder

Can you load metrics.log into vi and search there?

I just ssh'd into one of our forwarders and it worked for me...

$ grep -i "name=thruput" metrics.log
01-26-2017 15:16:23.841 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.202834, instantaneous_eps=0.257975, average_kbps=19.430535, total_k_processed=4248380.000000, kb=6.290039, ev=8.000000, load_average=2.010000
01-26-2017 15:16:54.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.219149, instantaneous_eps=0.613138, average_kbps=19.427809, total_k_processed=4248386.000000, kb=6.791016, ev=19.000000, load_average=1.610000
01-26-2017 15:17:25.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.209961, instantaneous_eps=0.258064, average_kbps=19.425083, total_k_processed=4248392.000000, kb=6.508789, ev=8.000000, load_average=1.500000
0 Karma
Get Updates on the Splunk Community!

Splunk Smartness with Brandon Sternfield | Episode 3

Hello and welcome to another episode of "Splunk Smartness," the interview series where we explore the power of ...

Monitoring Postgres with OpenTelemetry

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power ...

Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In ...