Create charts from SNMP Counter data type

[rasingh]

I am collecting snmpget data from a SAN switch. A few of the SNMP elements use counters where I get the accumalated value on each SNMPget. We want to graph the value of the difference between the last value and the current value of that counter over time. I think a new field would need to be created to show this difference before graphing.

This would be similar to say graphing port throughput in Cacti/MRTG/RRDtool.

[gkanapathy · Accepted solution]

You can use the | delta search command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Delta or in more complex cases the | streamstats command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Streamstats

[lukeh · Accepted solution]

We managed to get it working 🙂

We are using this splunk search with Network Traffic statistics sourced from Nagios Plugin Performance Data :-

index=nagios plugin="Network Traffic" hostname="server01"
| sort _time
| delta inAbsolut as inabs_change
| delta outAbsolut as outabs_change
| where inabs_change>=0
| where outabs_change>=0
| eval inkilobits=(inabs_change*8/1000)
| eval outkilobits=(outabs_change*8/1000)
| timechart span=5m per_second(inkilobits) as in_kbps, per_second(outkilobits) as out_kbps

Remember to select Null values: 'Connect' when displaying the above Powerful Graph after clicking "Show report" 🙂

Note: you need to include the 'where' statements to exclude negative delta values (caused by counter resets)

I will be including the above Powerful Graph when I release Splunk for Nagios v. 1.1 in Oct '10, as well as powerful graphs of other metal level metrics (cpu, memory, swap, load, disk usage, processes, etc) sourced from Nagios Plugin Performance Data 🙂

[rasingh]

I just tested and noticed a couple of things:
1. The 'where' statements should replace '>0' with '>=0' to remove negative delta values.
2. In comparing our Cacti/RRDtool graphs with splunk graphs, we selected "Connect" instead of "Treat as zero" on the graph to make them the same.

[rasingh]

I have tried generating the graphs but they do not seem accurate. What concerns me right now is that the counters do not operate as I think they do. When I do delta on TXwords say, I get negative numbers. That whould only happen if counter value resets.

[lukeh]

I am also looking to do the same thing, could you please share the search that you used to generate the requisite powerful graph? 😛

[rasingh]

First, I installed net-snmp and the switch MIBs (Brocade in this case). I use the 'snmptable' command to query all the ports at once and output into csv format - there are several command line options for that.

Once the command polling is tested, I created a batch file in $SPLUNK_HOME/bin as a wrapper to the snmptable command in the install location.

Lastly, I added the batch file with parameters as a scripted input.

[Dan]

Can you share how you are doing the snmp polling and putting the events into Splunk?

[gkanapathy · Accepted solution]

You can use the | delta search command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Delta or in more complex cases the | streamstats command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Streamstats