Confirming Cisco Networks Apps are working properly

[mbruce9]

I have Splunk 6.1 and a small Cisco network (10 network devices, 2 x WLC with 50 APs) and have installed both the Cisco networks Add on and Cisco Networks App; and am not quite clear if it is all working properly despite reviewing and trying to follow the App Help.
The Cisco Networks app is sparsely populated with data with > 50% unpopulated. Not sure if this is because this is only the data is available or what.

Here is a summary of the relevant splunk config shows:
- both apps installed (and reboot performed)
- Data Inputs (syslog data ) are configured as UDP Port 514 (the App instructions say choose another port but this was done previously to app install) - with sourcetype 'syslog' - no option appears for cisco:ios
- Sourcetype Cisco:SmartCallHome is set for TCP input on port 8989
- Switches have had their configurations set as shown in help (for 3750x all the commands are supported; not all features are supported for some others).

Results:
- from the base search interface, the range of fields (src_interface,mnemonic, message text....) and the sourcetype = cisco:ios suggest logs are being properly indexed with cisco relevant interpretations
- in the Cisco networks APP, most of the panels are not populated as if data is not available. No inventory info (although ATHome is believed to be configured and working), site names etc. NOt clear how these names are set.
- in The switching drop down, there is some data
- Wireless - empty panels. Not clear if every AP needs to send logging info or just WLCs (we have just configured the latter)

Questions:
a) How is the Cisco Add On - 'called up / specified' for the data inputs as above? Seems to be working; but perhaps I don't understand how the indexing rules and fields are determined as being applicable (though it seems to be working)
b) How does one debug a data input - to see what data is being received to enable troubleshooting
c) Wireless - any ideas of what makes this panel produce a useful overview?

[mbruce9]

Firstly thanks for your response. This has taken me longer than I should to acknowledge your assistance and address your points.

• Call Home - we haven't resolved this and this will be the issue with the site name
• otherwise it seems good.

To your point c) we have done more work to try and answer this. I found this reference on the web which gave some ideas
WLC Syslog Analysis

In trying to understand what was happening we came up with something similiar as follows:
a) Top WLC Syslog Events
search string: host="WLC*" message_text="*" |timechart span=30m count(mnemonic) by mnemonic
==> This is similiar to your "Top mnemonics by time" and This gives the stack of syslog events and one can see which of them is most common currently and the abstracted name. The stack is slightly more readable than the chart in my view.
In general
- for most syslog messages from the WLC - most of these typically look like bugs. (The network / switching layer produces more interesting and actionable messages).
- Ideally one would process some of the alarms in the same way to have a more useful Splunk view of all the alarms - but I haven't see any easy way to do this. In theory it is all possible.

Other general comments:
Switching dashboard / spanning tree and mac flapping. These are good, but we found a graph of when it occurred over time was useful. At a glance when things aren't going well; someone doesn't have to read times and dates, the graph says what is happening (30day default period )
search string: eventtype="cisco_ios-spanning_tree" | timechart count
eventtype="cisco_ios-mac_flapping" | timechart count

[mikaelbje]

Thanks for your suggestions. I've included the timecharts for spanning tree and MAC flapping in the dashboard in my development repository which means this will be included in the next public release of the app. I will also look into the mnemonic timechart for WLC. Looks like the only way I can know for sure that a host is a WLC is to do a search based on filename=* (a field that is only extracted from WLC events)

[mikaelbje]

Hi!

First of I just want to mention that Catalyst 2960 and other "lower end" switches are not able to send periodic Smart Call Home updates. This is stated in the Cisco docs. For 3750+ periodic updates will work.

Site names: This is set through the site-name "blabla" in the Call Home profile on the device.
Wireless: You can configure the APs to log directly, but they only send logs when they're autonomous or are not connected to a WLC. I have not tested this thoroughly. The current functionality reads logs from the WLCs

a) The app searches for two sourcetypes: cisco:ios and Cisco:SmartCallHome. No index name is specified in the base search, so if you are storing your logs in an index other than the main/default index you need to change the permissions of the Splunk role your user has to search that index by default.

b) Do a search for: index=* sourcetype=cisco:ios . You can see if the amount of events received are what you're expecting. The same goes for sourcetype Cisco:SmartCallHome

c) My main area is Routing and Switching, so this was one of the toughest views to create as I am not sure what a Wireless guy wants to see. Most of the panels in this dashboard look for MAC addresses in WLC events and try to show something useful based on that. WLCs are very chatty when it comes to logging, and I think a lot of the stuff being logged is junk.

Let me know if you have other questions.