Getting the error "Streamed search execute failed because: vector::_M_range_check" when doing any search that contains logs from a specific make of router

[tegnatomm]

We have been having an issue with the Cisco IOS Add-on installed on a search head returning logs from a specific router. When we do any search that returns results from this one router, we get the error: "Streamed search execute failed because: vector::_M_range_check" from all indexers.

The search is simple: sourcetype="cisco:ios" over some time period that contains data from the device.

The problem though does not seem to be the indexers. Searches work with no errors once again if we disable the Cisco Networks Add-on for Splunk Enterprise on the search head. The documentation from the Cisco Networks App for Splunk Enterprise says it needs this add-on installed on all indexers and search heads. We have done that.

The router in question is a Cisco WS-C4500X-32.

Here is an example of the raw syslog data from this router that is causing issues:

2016 Jun  1 11:25:13 -04:00 192.168.64.1 Jun  1 07:25:09.822 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.81.10 (Vlan412) is up: new adjacency
2016 Jun  1 11:25:13 -04:00 192.168.64.1 Jun  1 07:25:09.948 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.81.6 (Vlan411) is up: new adjacency
2016 Jun  1 11:25:14 -04:00 192.168.64.1 Jun  1 07:25:11.245 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.97.17 (Port-channel82) is up: new adjacency
2016 Jun  1 11:25:14 -04:00 192.168.64.1 Jun  1 07:25:11.330 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.97.13 (Port-channel81) is up: new adjacency

Any thoughts or ideas about this one or directions to help troubleshoot this?

[mikaelbje]

This is a known bug in Splunk 6.3.0. Upgrade to the latest 6.3 release or 6.4 release.

The documentation of the Cisco Networks app and add-on contains info about this, so make sure you read the documentation.