All Apps and Add-ons

Cisco Networks Add-on for Splunk Enterprise: How to get wireless lan controller logs to parse correctly?

Explorer

Running version 2.2.1. I guess the readme.txt was never updated, but the README.md reflects the latest version. This is add-on is working great for regular cisco devices, but no such luck for our wireless lan controllers. The log format obviously doesn't match exactly to what's in samplelog.cisco.ios: I have it coming in as the same sourcetype cisco:ios. Should I just separate WLC to regular syslog?

Aug 19 11:50:36 HHHHHHHH 8427: Aug 19 11:50:35.221 CEST: %LDP-5-NBRCHG: LDP Neighbor XXX.XXX.XXX.XXX:0 (1) is DOWN (Interface not operational)

Looking at the transforms.conf in default for TA-cisco_ios:

## TODO: NEED TO ADD extract_cisco_ios-general-wlc (8500) and remove the # extraction above. Because time ends up in reported_hostname

Is the exact problem I'm still having... So what am I missing? Were changes actually implemented? Did I miss a step somewhere? Transforms needs to be moved to local maybe, being overwritten by my system transforms, yet looking at the app default transform, it reads like it's broken as expected. :facepalm:

0 Karma

Motivator

Hi!

  1. Do you get WLC events as sourcetype cisco:ios ?
  2. Are the fields facility and mnemonic properly extracted?
  3. Can you send me some raw log samples as seen in your Splunk instance?

The TODO item never made it in, but WLC events do get properly indexed on several installations I have. The TODO item is just cosmetic/refinements

Mikael

0 Karma

Explorer
  1. Yes their sourcetype is cisco:ios
  2. facility and mnemonic are both being extracted properly. The product is being extracted as WLC too

Some examples with hostnames and ip addresses removed:

Aug 4 14:46:16 FQDN SHORTNAME: *ethoipSocketTask: Aug 04 14:46:16.447: %ETHOIP-3-PKTRECVERROR: ethoip.c:338 ethoipSocketTask: ethoipRecvPkt returned error
Aug 4 14:46:16 FQDN SHORTNAME: *ethoipSocketTask: Aug 04 14:46:16.446: %ETHOIP-4-RECVDPKTFROMNONMEMBER: ethoip.c:131 Recv Ethernet over IP ping from x.x.x.x, not from a mobility peer
Aug 4 14:46:13 FQDN SHORTNAME: *mmListen: Aug 04 14:46:13.874: %MM-3-INVALIDPKTRECVD: mm_listen.c:7671 Received an invalid packet from x.x.x.x. Source member:0.0.0.0. source member unknown.

If there is confusion, the events are being indexed, just with the time stamp being extracted as hostname just as the TODO comment states.

0 Karma

Motivator

Ok, thanks for confirming this. Do you still have the correct hostname in the host field? I'll look into this at some point, but can't really promise anything. The app is still usable with WLC, but not the reported_hostname and device_time fields I guess?

Getting this right would be involve quite a lot of fiddling with regular expressions. To further complicate matters Cisco decided to change the logging format from using % to # before the facility field in a WLC release.

To sum it up, this part is broken as expected. You should be fine with WLC events in the cisco:ios sourcetype. The other extractions such as facility, severityid, mnemonic and all the lookups should are way more useful than the reportedhostname and device_time fields.

0 Karma

Explorer

No worries just wanted to make sure it was expected. I wish the correct hostname was being picked up. The host field is being populated by the time, just as the CiscoIOSEvent.reported_hostname is.

But yea the rest of the events are extracted properly, just not tied to any device since it's all a different HH:MM:SS

0 Karma

Motivator

Aha! Are your logs sent directly to Splunk or do you have a syslog daemon saving the logs to file? A common setup is to use i.e. syslog-ng or rsyslog and save each host to its own folder like this: /var/log/remote/$HOST/syslog

You then set up Splunk to monitor those directories and set host_segment=4 and sourcetype=syslog or sourcetype=cisco:ios for the input. This way you'll get the right hostname instead of relying on Splunk to pick up the hostname from the right position in the event.

It would probably work the same way if you had the sourcetype set to syslog for the Splunk UDP input too, but Splunk would have to apply the transform to every event coming in as syslog to check if it should be changed to cisco:ios.

If in fact the latter method works you could just take the default host transform for syslog from $SPLUNKHOME/etc/system/default/props.conf and apply that to cisco:ios in $SPLUNKHOME/etc/apps/TA-cisco_ios/local/props.conf

Explorer

So yes and yes. The majority of devices out there have the uf installed and sending direct. However our cisco gear is sending to linux rsyslog boxes, and in this case being ingested with the sourcetype defined as cisco:ios

Not currently setting the host_segment on that monitored directory. I think the vast majority follow that, maybe not meraki? Will find out. That's probably it. And if so, thanks for the help. 😛 Was just dreading the prospect of sending our WLC's to a different destination special for them. Will report back and mark as answer in a bit 😉

0 Karma

Motivator

Did you test this?

props.conf:
[cisco:ios]
TRANSFORMS = syslog-host

0 Karma