Running version 2.2.1. I guess the readme.txt was never updated, but the README.md reflects the latest version. This is add-on is working great for regular cisco devices, but no such luck for our wireless lan controllers. The log format obviously doesn't match exactly to what's in samplelog.cisco.ios: I have it coming in as the same sourcetype cisco:ios. Should I just separate WLC to regular syslog?
Aug 19 11:50:36 HHHHHHHH 8427: Aug 19 11:50:35.221 CEST: %LDP-5-NBRCHG: LDP Neighbor XXX.XXX.XXX.XXX:0 (1) is DOWN (Interface not operational)
Looking at the transforms.conf in default for TA-cisco_ios:
## TODO: NEED TO ADD extract_cisco_ios-general-wlc (8500) and remove the # extraction above. Because time ends up in reported_hostname
Is the exact problem I'm still having... So what am I missing? Were changes actually implemented? Did I miss a step somewhere? Transforms needs to be moved to local maybe, being overwritten by my system transforms, yet looking at the app default transform, it reads like it's broken as expected. :facepalm:
The TODO item never made it in, but WLC events do get properly indexed on several installations I have. The TODO item is just cosmetic/refinements
Some examples with hostnames and ip addresses removed:
Aug 4 14:46:16 FQDN SHORTNAME: *ethoipSocketTask: Aug 04 14:46:16.447: %ETHOIP-3-PKTRECVERROR: ethoip.c:338 ethoipSocketTask: ethoipRecvPkt returned error
Aug 4 14:46:16 FQDN SHORTNAME: *ethoipSocketTask: Aug 04 14:46:16.446: %ETHOIP-4-RECVDPKTFROMNONMEMBER: ethoip.c:131 Recv Ethernet over IP ping from x.x.x.x, not from a mobility peer
Aug 4 14:46:13 FQDN SHORTNAME: *mmListen: Aug 04 14:46:13.874: %MM-3-INVALIDPKTRECVD: mm_listen.c:7671 Received an invalid packet from x.x.x.x. Source member:0.0.0.0. source member unknown.
If there is confusion, the events are being indexed, just with the time stamp being extracted as hostname just as the TODO comment states.
Ok, thanks for confirming this. Do you still have the correct hostname in the host field? I'll look into this at some point, but can't really promise anything. The app is still usable with WLC, but not the reported_hostname and device_time fields I guess?
Getting this right would be involve quite a lot of fiddling with regular expressions. To further complicate matters Cisco decided to change the logging format from using % to # before the facility field in a WLC release.
To sum it up, this part is broken as expected. You should be fine with WLC events in the cisco:ios sourcetype. The other extractions such as facility, severityid, mnemonic and all the lookups should are way more useful than the reportedhostname and device_time fields.
No worries just wanted to make sure it was expected. I wish the correct hostname was being picked up. The host field is being populated by the time, just as the CiscoIOSEvent.reported_hostname is.
But yea the rest of the events are extracted properly, just not tied to any device since it's all a different HH:MM:SS
Aha! Are your logs sent directly to Splunk or do you have a syslog daemon saving the logs to file? A common setup is to use i.e. syslog-ng or rsyslog and save each host to its own folder like this: /var/log/remote/$HOST/syslog
You then set up Splunk to monitor those directories and set host_segment=4 and sourcetype=syslog or sourcetype=cisco:ios for the input. This way you'll get the right hostname instead of relying on Splunk to pick up the hostname from the right position in the event.
It would probably work the same way if you had the sourcetype set to syslog for the Splunk UDP input too, but Splunk would have to apply the transform to every event coming in as syslog to check if it should be changed to cisco:ios.
If in fact the latter method works you could just take the default host transform for syslog from $SPLUNKHOME/etc/system/default/props.conf and apply that to cisco:ios in $SPLUNKHOME/etc/apps/TA-cisco_ios/local/props.conf
So yes and yes. The majority of devices out there have the uf installed and sending direct. However our cisco gear is sending to linux rsyslog boxes, and in this case being ingested with the sourcetype defined as cisco:ios
Not currently setting the host_segment on that monitored directory. I think the vast majority follow that, maybe not meraki? Will find out. That's probably it. And if so, thanks for the help. 😛 Was just dreading the prospect of sending our WLC's to a different destination special for them. Will report back and mark as answer in a bit 😉