Documented recipe for replicating subsets of data to 3rd party system throws errors in 4.3.1 and 4.3.2

[Wiggy]

In 4.3.1 and 4.3.2, when using the documented procedure to replicate subsets of data to 3rd party systems found here:

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad#Replicate_a_subset_of_...

and using the following code recipe in $splunk_home/etc/system/local/outputs.conf:

[tcpout]
defaultGroup = nothing

[tcpout:firstlocation]
disabled = false
server = foo.bar1.com:9998
sendCookedData = false
indexAndForward = true

[tcpout:secondlocation]
disabled = false
server = foo.bar2.com:9998
sendCookedData=false
indexAndForward = true

splunk creates a large amount of errors in splunkd.log in this format:

04-24-2012 09:14:44.078 -0500 ERROR splunklogger - Uncaught exception in pipeline execution (tcp-output-generic-processor) - getting next event
04-24-2012 09:14:44.078 -0500 ERROR pipeline - Runtime exception in pipeline: indexerPipe, processor: tcp-output-generic-processor, error: vector::_M_range_check
04-24-2012 09:14:44.078 -0500 ERROR splunklogger - Uncaught exception in pipeline execution (tcp-output-generic-processor) - getting next event
04-24-2012 09:14:44.078 -0500 ERROR pipeline - Runtime exception in pipeline: indexerPipe, processor: tcp-output-generic-processor, error: vector::_M_range_check

and results in the data not being written to any index.

[Wiggy · Accepted solution]

This is a known bug in 4.3.1 and 4.3.2. The bug is SPL-50576 and has been fixed and will be released with the 4.3.3 version of splunk. In the mean-time, the work around is as follows:

You must define a [tcpout:nothing], give it a false address, and tell it to drop events on queue full as shown in below example:

[tcpout]
defaultGroup = nothing

[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
indexAndForward = true 
dropEventsOnQueueFull = 1

[tcpout:firstlocation]
disabled = false
server = foo.bar1.com:9998
sendCookedData = false
indexAndForward = true

[tcpout:secondlocation]
disabled = false
server = foo.bar2.com:9998
sendCookedData=false
indexAndForward = true

This will circumvent the problem with routing to a non-existant group and index all data as normal, also routing any other data you have set up to go to the other output stanzas as expected.