*Nix App - Network Throughput Calculations

[jdunlea_splunk]

Hi Guys,

I have some confusions around the Interface Throughput calculations.

The following search seems to be finding the average of the DIFFERENCE between the last TX value and the current TX value. What are the TX values representing? The current upload bytes for that poll period, or the accumulated upload bytes for that interface?

index="os" sourcetype="interfaces" host=* | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) by Interface_Host

What are we trying to calculate here? Also, is this an accurate representation of bandwidth usage for that interface on a system?

Cheers,

John

[jodros]

I have also noticed some weirdness with this charting.

First, it appears as if you are grouping all of your streamstats calculations by only Name. I believe it should be by "host Name". If you don't first group by host and then Name, you wind up calculating all "eth0" stats across your entire environment, which isn't something I believe you are trying to do.

Second, you take the time to get the receive Kb, but then do not chart it. Why even worry about RX if you don't chart it? I personally would like to see the RX numbers as well.

I have modified the search to the following and have found much more accurate results:

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | timechart eval(sum(TX_Thruput_KB)/dc(time)) as TX eval(sum(RX_Thruput_KB)/dc(time)) as RX by Interface_Host

I also modified the XML to add Kilobytes to the y axis of the chart, since it did not notate the units used. Had to change it to "charting.secondaryAxisTitle.text" for KB to show.

[pollockm]

The two example above work great, but I'm wondering if someone could elaborate on part of them. In the Thrupart Chart statement, at the end there is eval(sum(RX_Thruput_KB)/dc(time)). I don't understand why you are dividing by dc(time). Does it have something to do with setting the span in the timechart?

[lemikg]

@jodros: perfect, thanks a lot.

[jodros]

Top Interfaces Table

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | search RX_Thruput_KB OR TX_Thruput_KB >= 0 | stats sum(TX_Thruput_KB) as "Total KB Transmitted" sum(RX_Thruput_KB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20

[jodros]

Thruput Chart

index="os" sourcetype="interfaces" host=ho-splunkds1 | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | search RX_Thruput_KB OR TX_Thruput_KB >= 0 | timechart eval(sum(TX_Thruput_KB)/dc(time)) as TX eval(sum(RX_Thruput_KB)/dc(time)) as RX by Interface_Host

[jodros]

@lemikg, I finally encountered an instance where this chart displayed negative integers. This was due to the server rebooting. Due to the way the search is calculating thruput, servers rebooting, splunk services stopping for a period of time, etc, can cause those negative integers to appear in charting. In order for the charts to be accurate, they need to have had the search run at least twice normally after all server related issues are corrected. An easy fix for this is to search for only positive integers to chart. I have modified the searches below. Thanks.

[jodros]

@lemikg, are you still having issues with the chart?

[lemikg]

@jodros, no I didn't.

[jodros]

@lemikg, that is unusual. I just tested and I am not experiencing that behavior, either with the negative values or with the differing peak times.

Did you modify the search in anyway in your environment?

[lemikg]

Hi, i am referring to interface throughput. And also i noticed, that The time of The events doesnt match up if i go from "all hosts" to a selected one. For example in The overall chart host A peaks at 4 pm and when i select just that host it shows that The event occured at 5:30 am.

[jodros]

@lemikg, which search are you referring, the Interface Throughput, or the Top Interfaces?

Thanks

[lemikg]

@jodros: thank you very much for the modified search. However, I am getting negative results. Did anybody experience such an outcome and help me with some insight?

cheers
Mike

[jodros]

index="os" sourcetype="interfaces" host=$host$ | multikv fields name, inetAddr, RXbytes, TXbytes | streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by host Name | eval time=_time | strcat Name "-" inetAddr "@" host Interface_Host | eval RX_Thruput_KB = (lastRX-RXbytes)/1024 | eval TX_Thruput_KB = (lastTX-TXbytes)/1024 | stats sum(TX_Thruput_KB) as "Total KB Transmitted" sum(RX_Thruput_KB) as "Total KB Received" by Interface_Host | sort -"Total KB Received" | head 20

[jodros]

I wanted to update with my modified "Top Interfaces" table. Basically showing total KB tx and rx during the time selected on the dropdown. The search is below:

[araitz]

I'm not sure how that chart could be useful either 🙂 Thanks for your modified search, we will take a look at it and try to incorporate lessons from it back in to the app.

[jodros]

Also, can someone please explain to me what the "Top Interfaces" chart powered by the Top_Inet_Addresses_by_Host search is supposed to be reporting? I find no usefulness in this chart currently.

Thanks

[tiberious726]

Accumulated total bytes, just like ifconfig/the ip2 suite (Btw, that search is much more easy if you use the delta search command)

[tiberious726]

just "|delta TX" it will save the values in a field called delta(TX)

[lemikg]

hi @tiberious726,
do you have an example query with the delta command?
cheers, Mike

[tiberious726]

The TX value is accumulated total bytes, and yes, that is why you are finding the difference (Tho I would look at the "delta" command, it does that too and would probably be much more efficient).

This command is pulling out the fields "streamstats current=f last(TXbytes) as lastTX, last(RXbytes) as lastRX by Name", It should be pulling them out in order, I'm not sure why it wouldnt (which would yield negative numbers). Try making sure that the "by Interface_Host" is actually working.

Try looking at the raw data and make sure the tx values are increasing relative to the time stamp.