topic Re: *Nix App - Network Throughput Calculations in All Apps and Add-ons

*Nix App - Network Throughput Calculations

jdunlea_splunk — Wed, 20 Feb 2013 22:01:25 GMT

Hi Guys,

I have some confusions around the Interface Throughput calculations.

The following search seems to be finding the average of the DIFFERENCE between the last TX value and the current TX value. What are the TX values representing? The current upload bytes for that poll period, or the accumulated upload bytes for that interface?

index="os" sourcetype="interfaces" host=* | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) by Interface_Host

What are we trying to calculate here? Also, is this an accurate representation of bandwidth usage for that interface on a system?

Cheers,

John

Re: *Nix App - Network Throughput Calculations

tiberious726 — Wed, 20 Feb 2013 22:25:57 GMT

Accumulated total bytes, just like ifconfig/the ip2 suite (Btw, that search is much more easy if you use the delta search command)

Re: *Nix App - Network Throughput Calculations

jdunlea_splunk — Wed, 20 Feb 2013 22:36:06 GMT

Hi tiberious726,

So you are saying that the SEARCH is calculating the "accumulated total bytes", or that the straight TX value in the events is the "accumulated total bytes" (so that is why we are finding the difference between TXbytes and lastTX in this search)? The latter makes the most sense to me. What is strange , is that for some of my instances, I am seeing negative results for the different between the current TXbytes value and the lastTX.... which does not make any sense?

Re: *Nix App - Network Throughput Calculations

tiberious726 — Thu, 21 Feb 2013 03:33:59 GMT

The TX value is accumulated total bytes, and yes, that is why you are finding the difference (Tho I would look at the "delta" command, it does that too and would probably be much more efficient).

This command is pulling out the fields "streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name", It should be pulling them out in order, I'm not sure why it wouldnt (which would yield negative numbers). Try making sure that the "by Interface_Host" is actually working.

Try looking at the raw data and make sure the tx values are increasing relative to the time stamp.

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 18 Mar 2013 13:56:47 GMT

I have also noticed some weirdness with this charting.

First, it appears as if you are grouping all of your streamstats calculations by only Name. I believe it should be by "host Name". If you don't first group by host and then Name, you wind up calculating all "eth0" stats across your entire environment, which isn't something I believe you are trying to do.

Second, you take the time to get the receive Kb, but then do not chart it. Why even worry about RX if you don't chart it? I personally would like to see the RX numbers as well.

I have modified the search to the following and have found much more accurate results:

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) as TX eval(sum(RX_Thruput_KB)/dc(time)) as RX by Interface_Host

I also modified the XML to add Kilobytes to the y axis of the chart, since it did not notate the units used. Had to change it to "charting.secondaryAxisTitle.text" for KB to show.

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 28 Sep 2020 13:32:26 GMT

Also, can someone please explain to me what the "Top Interfaces" chart powered by the Top_Inet_Addresses_by_Host search is supposed to be reporting? I find no usefulness in this chart currently.

Thanks

Re: *Nix App - Network Throughput Calculations

araitz — Mon, 18 Mar 2013 20:28:22 GMT

I'm not sure how that chart could be useful either 🙂 Thanks for your modified search, we will take a look at it and try to incorporate lessons from it back in to the app.

Re: *Nix App - Network Throughput Calculations

jodros — Tue, 19 Mar 2013 20:04:47 GMT

I wanted to update with my modified "Top Interfaces" table. Basically showing total KB tx and rx during the time selected on the dropdown. The search is below:

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 28 Sep 2020 13:32:50 GMT

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | stats sum(TX_Thruput_KB) as "Total KB Transmitted" sum(RX_Thruput_KB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20

Re: *Nix App - Network Throughput Calculations

lemikg — Fri, 22 Mar 2013 16:37:03 GMT

@jodros: thank you very much for the modified search. However, I am getting negative results. Did anybody experience such an outcome and help me with some insight?

cheers
Mike

Re: *Nix App - Network Throughput Calculations

jodros — Fri, 22 Mar 2013 16:41:10 GMT

@lemikg, which search are you referring, the Interface Throughput, or the Top Interfaces?

Thanks

Re: *Nix App - Network Throughput Calculations

lemikg — Fri, 22 Mar 2013 17:03:04 GMT

Hi, i am referring to interface throughput. And also i noticed, that The time of The events doesnt match up if i go from "all hosts" to a selected one. For example in The overall chart host A peaks at 4 pm and when i select just that host it shows that The event occured at 5:30 am.

Re: *Nix App - Network Throughput Calculations

jodros — Fri, 22 Mar 2013 17:46:45 GMT

@lemikg, that is unusual. I just tested and I am not experiencing that behavior, either with the negative values or with the differing peak times.

Did you modify the search in anyway in your environment?

Re: *Nix App - Network Throughput Calculations

lemikg — Sat, 23 Mar 2013 18:47:36 GMT

@jodros, no I didn't.

Re: *Nix App - Network Throughput Calculations

lemikg — Sat, 23 Mar 2013 18:50:11 GMT

hi @tiberious726,
do you have an example query with the delta command?
cheers, Mike

Re: *Nix App - Network Throughput Calculations

tiberious726 — Sat, 23 Mar 2013 23:06:14 GMT

just "|delta TX" it will save the values in a field called delta(TX)

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 08 Apr 2013 13:15:33 GMT

@lemikg, are you still having issues with the chart?

Re: *Nix App - Network Throughput Calculations

jodros — Wed, 10 Apr 2013 13:18:03 GMT

@lemikg, I finally encountered an instance where this chart displayed negative integers. This was due to the server rebooting. Due to the way the search is calculating thruput, servers rebooting, splunk services stopping for a period of time, etc, can cause those negative integers to appear in charting. In order for the charts to be accurate, they need to have had the search run at least twice normally after all server related issues are corrected. An easy fix for this is to search for only positive integers to chart. I have modified the searches below. Thanks.

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 28 Sep 2020 13:42:28 GMT

Thruput Chart

index="os" sourcetype="interfaces" host=ho-splunkds1 | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | search RX_Thruput_KB OR TX_Thruput_KB >= 0 | timechart eval(sum(TX_Thruput_KB)/dc(time)) as TX eval(sum(RX_Thruput_KB)/dc(time)) as RX by Interface_Host

Re: *Nix App - Network Throughput Calculations

jodros — Mon, 28 Sep 2020 13:42:31 GMT

Top Interfaces Table

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | search RX_Thruput_KB OR TX_Thruput_KB >= 0 | stats sum(TX_Thruput_KB) as "Total KB Transmitted" sum(RX_Thruput_KB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20