topic Re: Create charts from SNMP Counter data type in Splunk Search

Create charts from SNMP Counter data type

rasingh — Fri, 09 Apr 2010 03:43:56 GMT

I am collecting snmpget data from a SAN switch. A few of the SNMP elements use counters where I get the accumalated value on each SNMPget. We want to graph the value of the difference between the last value and the current value of that counter over time. I think a new field would need to be created to show this difference before graphing.

This would be similar to say graphing port throughput in Cacti/MRTG/RRDtool.

Re: Create charts from SNMP Counter data type

gkanapathy — Fri, 09 Apr 2010 04:36:58 GMT

You can use the | delta search command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Delta or in more complex cases the | streamstats command: http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Streamstats

Re: Create charts from SNMP Counter data type

Dan — Fri, 09 Apr 2010 19:09:14 GMT

Can you share how you are doing the snmp polling and putting the events into Splunk?

Re: Create charts from SNMP Counter data type

rasingh — Tue, 13 Apr 2010 02:54:00 GMT

First, I installed net-snmp and the switch MIBs (Brocade in this case). I use the 'snmptable' command to query all the ports at once and output into csv format - there are several command line options for that.

Once the command polling is tested, I created a batch file in $SPLUNK_HOME/bin as a wrapper to the snmptable command in the install location.

Lastly, I added the batch file with parameters as a scripted input.

Re: Create charts from SNMP Counter data type

lukeh — Thu, 02 Sep 2010 15:32:14 GMT

I am also looking to do the same thing, could you please share the search that you used to generate the requisite powerful graph? 😛

Re: Create charts from SNMP Counter data type

rasingh — Wed, 08 Sep 2010 07:09:14 GMT

I have tried generating the graphs but they do not seem accurate. What concerns me right now is that the counters do not operate as I think they do. When I do delta on TXwords say, I get negative numbers. That whould only happen if counter value resets.

Re: Create charts from SNMP Counter data type

lukeh — Thu, 16 Sep 2010 11:33:03 GMT

We managed to get it working 🙂

We are using this splunk search with Network Traffic statistics sourced from Nagios Plugin Performance Data :-

index=nagios plugin="Network Traffic" hostname="server01"
| sort _time
| delta inAbsolut as inabs_change
| delta outAbsolut as outabs_change
| where inabs_change>=0
| where outabs_change>=0
| eval inkilobits=(inabs_change*8/1000)
| eval outkilobits=(outabs_change*8/1000)
| timechart span=5m per_second(inkilobits) as in_kbps, per_second(outkilobits) as out_kbps

Remember to select Null values: 'Connect' when displaying the above Powerful Graph after clicking "Show report" 🙂

Note: you need to include the 'where' statements to exclude negative delta values (caused by counter resets)

I will be including the above Powerful Graph when I release Splunk for Nagios v. 1.1 in Oct '10, as well as powerful graphs of other metal level metrics (cpu, memory, swap, load, disk usage, processes, etc) sourced from Nagios Plugin Performance Data 🙂

Re: Create charts from SNMP Counter data type

rasingh — Fri, 17 Sep 2010 05:46:32 GMT

I just tested and noticed a couple of things:
1. The 'where' statements should replace '>0' with '>=0' to remove negative delta values.
2. In comparing our Cacti/RRDtool graphs with splunk graphs, we selected "Connect" instead of "Treat as zero" on the graph to make them the same.