How to search how much bandwidth a forwarder is using?

[sbattista09]

I'm trying to find how much bandwidth a forwarder is using and how many hosts are sending over the forwarder. I want to show it in a timechart that has the hosts' total bandwidth and then another line that had all the totals. I'm not sure where to start since most documents show using the _internal index. any input will help, thanks!

[jbsplunk · Accepted solution]

By default, the universal forwarder sends some data to the indexers, and one of these logs is the metrics.log, which you can use to calculate the outgoing thruput rate every 30s.

The way this is done in Splunk on Splunk(you can simply look at the metrics view if you've got that app installed) is like something like this:

index=_internal source=*metrics.log* group=tcpout_connections | eval KBps=tcp_Bps/1024 | timechart sum(KBps) as KBps span=15m by host| eval MB=KBps*_span/1024 | fields - KBps

[louismai]

index=_internal source=metrics.log group=udpin_connections |timechart avg(udp_kbps) max(udp_kbps) span=5m

UDP is similar. But UDP has _udp_bps=25434.085277542734, _udp_kbps=24.837973903850326, _udp_avg_thruput=15.692977114824794; so we might not need to do calculation.

[twinspop]

You risk double accounting the metrics being logged ON the indexer and the metrics being logged TO the indexer (ON the forwarders). I would encourage you to include a host qualifier in your search.

[jkellerman_splu]

Wouldn't avg(KBps) be the correct function here?
Otherwise you would be presenting the sum of the individual measurements during the 15 min span. That would not be per second.

[sbattista09]

jkellerman,
can you show me what you mean, is that not what is in my search?

[lqiao]

Hi sbattista09,

I think jkellerman was referring the answer from jbsplunk that the function in the timechart should be avg instead of sum which I agree.

[sowings]

Splunk on Splunk? Don't you mean DMC? 🙂

[sbattista09]

awesome!

I also found this nifty little search-

| rest splunk_server=* /services/server/introspection/indexer | eval status = case(reason == ".", status,   reason == "", status,   isnull(reason), status,   1 == 1, status.": ".reason) | fields splunk_server, average_KBps, status   | eval average_KBps = round(average_KBps, 0) | rename splunk_server as Instance, average_KBps as "Indexing Rate (KB/s)", status as Status, reason as Reason