All Apps and Add-ons

*Nix App - Network Throughput Calculations

Splunk Employee
Splunk Employee

Hi Guys,

I have some confusions around the Interface Throughput calculations.

The following search seems to be finding the average of the DIFFERENCE between the last TX value and the current TX value. What are the TX values representing? The current upload bytes for that poll period, or the accumulated upload bytes for that interface?

index="os" sourcetype="interfaces" host=* | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) by Interface_Host

What are we trying to calculate here? Also, is this an accurate representation of bandwidth usage for that interface on a system?

Cheers,

John

0 Karma

Builder

I have also noticed some weirdness with this charting.

First, it appears as if you are grouping all of your streamstats calculations by only Name. I believe it should be by "host Name". If you don't first group by host and then Name, you wind up calculating all "eth0" stats across your entire environment, which isn't something I believe you are trying to do.

Second, you take the time to get the receive Kb, but then do not chart it. Why even worry about RX if you don't chart it? I personally would like to see the RX numbers as well.

I have modified the search to the following and have found much more accurate results:

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) as TX eval(sum(RX_Thruput_KB)/dc(time)) as RX by Interface_Host

I also modified the XML to add Kilobytes to the y axis of the chart, since it did not notate the units used. Had to change it to "charting.secondaryAxisTitle.text" for KB to show.

Engager

The two example above work great, but I'm wondering if someone could elaborate on part of them. In the Thrupart Chart statement, at the end there is eval(sum(RXThruputKB)/dc(time)). I don't understand why you are dividing by dc(time). Does it have something to do with setting the span in the timechart?

0 Karma

Communicator

@jodros: perfect, thanks a lot.

0 Karma

Builder

Top Interfaces Table

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=time | strcat Name "-" inetAddr "@" host InterfaceHost | eval RXThruputKB = (lastRX-RXbytes)/1024 | eval TXThruputKB = (lastTX-TXbytes)/1024 | search RXThruputKB OR TXThruputKB >= 0 | stats sum(TXThruputKB) as "Total KB Transmitted" sum(RXThruputKB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20

0 Karma

Builder

Thruput Chart

index="os" sourcetype="interfaces" host=ho-splunkds1 | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=time | strcat Name "-" inetAddr "@" host InterfaceHost | eval RXThruputKB = (lastRX-RXbytes)/1024 | eval TXThruputKB = (lastTX-TXbytes)/1024 | search RXThruputKB OR TXThruputKB >= 0 | timechart eval(sum(TXThruputKB)/dc(time)) as TX eval(sum(RXThruputKB)/dc(time)) as RX by Interface_Host

0 Karma

Builder

@lemikg, I finally encountered an instance where this chart displayed negative integers. This was due to the server rebooting. Due to the way the search is calculating thruput, servers rebooting, splunk services stopping for a period of time, etc, can cause those negative integers to appear in charting. In order for the charts to be accurate, they need to have had the search run at least twice normally after all server related issues are corrected. An easy fix for this is to search for only positive integers to chart. I have modified the searches below. Thanks.

Builder

@lemikg, are you still having issues with the chart?

0 Karma

Communicator

@jodros, no I didn't.

0 Karma

Builder

@lemikg, that is unusual. I just tested and I am not experiencing that behavior, either with the negative values or with the differing peak times.

Did you modify the search in anyway in your environment?

0 Karma

Communicator

Hi, i am referring to interface throughput. And also i noticed, that The time of The events doesnt match up if i go from "all hosts" to a selected one. For example in The overall chart host A peaks at 4 pm and when i select just that host it shows that The event occured at 5:30 am.

0 Karma

Builder

@lemikg, which search are you referring, the Interface Throughput, or the Top Interfaces?

Thanks

0 Karma

Communicator

@jodros: thank you very much for the modified search. However, I am getting negative results. Did anybody experience such an outcome and help me with some insight?

cheers
Mike

0 Karma

Builder

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=time | strcat Name "-" inetAddr "@" host InterfaceHost | eval RXThruputKB = (lastRX-RXbytes)/1024 | eval TXThruputKB = (lastTX-TXbytes)/1024 | stats sum(TXThruputKB) as "Total KB Transmitted" sum(RXThruputKB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20

0 Karma

Builder

I wanted to update with my modified "Top Interfaces" table. Basically showing total KB tx and rx during the time selected on the dropdown. The search is below:

0 Karma

Splunk Employee
Splunk Employee

I'm not sure how that chart could be useful either 🙂 Thanks for your modified search, we will take a look at it and try to incorporate lessons from it back in to the app.

0 Karma

Builder

Also, can someone please explain to me what the "Top Interfaces" chart powered by the TopInetAddressesbyHost search is supposed to be reporting? I find no usefulness in this chart currently.

Thanks

0 Karma

Path Finder

Accumulated total bytes, just like ifconfig/the ip2 suite (Btw, that search is much more easy if you use the delta search command)

0 Karma

Path Finder

just "|delta TX" it will save the values in a field called delta(TX)

0 Karma

Communicator

hi @tiberious726,
do you have an example query with the delta command?
cheers, Mike

0 Karma

Path Finder

The TX value is accumulated total bytes, and yes, that is why you are finding the difference (Tho I would look at the "delta" command, it does that too and would probably be much more efficient).

This command is pulling out the fields "streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name", It should be pulling them out in order, I'm not sure why it wouldnt (which would yield negative numbers). Try making sure that the "by Interface_Host" is actually working.

Try looking at the raw data and make sure the tx values are increasing relative to the time stamp.

0 Karma