Forwarder tcpout_connections blocked

[Chris_R_]

This configuration is two 3.4.2 forwarders -> two 4.1.2 indexers.
Forwarders have two UDP inputs & two seperate assigned sourcetypes on these UDP inputs, props/transforms/outputs entries are doing _TCP_ROUTING to two seperate indexers.
Config seems ok for the most part. However they are getting constantly blocked tcpout_connections messages in metrics.log

splunkd.log Error on the forwarders

07-07-2010 06:11:29.452 WARN  TcpOutputProc - TcpSendThread: Connection to server lost - retrying: Broken pipe  
07-07-2010 06:11:29.452 WARN  TcpOutputProc - Connection dropped by Indexer. Possible version mismatch with indexer. Please check compatibility with indexer version  

splunkd.log errors on the indexer

07-08-2010 01:15:13.501 ERROR TcpInputProc - Error encountered for connection from host=< ip address >, ip=< ip address >. Timeout  
07-08-2010 01:15:13.501 INFO  TcpInputProc - Hostname=< ip address > closed connection  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel source::udp:515|host::192.168.88.25|somesourcetypel|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:514|host::192.168.8.204|somesourcetypee|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.26|somesourcetype|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::/opt/splunk/var/log/splunk/splunklogger.log|host::NCCForwarder|splunklogger|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.27|somesourcetype|remoteport::41108" ended without a done-key  

one other odd entry i see in the inputs.conf of the indexers, seems like this is a older spec file setting to route certain data to queues instead of letting splunk do it automatically?

[splunktcp]
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;

[Chris_R_]

GK: These are full forwarders, here's the outputs from a forwarder

[tcpout]  
indexAndForward = false  


[tcpout:stonegateGroup]
disabled = false  
server=10.20.12.35:9001  

[tcpout:fortimailGroup]  
disabled = false  
server=10.20.12.33:9997  

and the inputs.conf from a indexer

[default]  
index = default  
host = fortimailsplunk  
_rcvbuf = 196608  

[monitor://$SPLUNK_HOME/var/spool/splunk]  
move_policy = sinkhole  

[fschange:$SPLUNK_HOME/etc]  
signedaudit = true  
sendEventMaxSize = -1  
recurse = true  
pollPeriod = 600  
filesPerDelay = 10  
delayInMills = 100  
followLinks = false  
fullEvent = false  
hashMaxSize = -1  

[splunktcp]  
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;  

Note: I had them remove the tcp route = stanza seems to not be blocking this morning, could be a slower day...but i'll know for sure next week

[gkanapathy]

That route is in fact in the etc/system/default/inputs.conf for 4.x machines. Someone might have copied it over. Don't mess with it.

Please clarify if these are heavy forwarders, or LWF's tweaked to collect UDP as well? It would be helpful to see the outputs.conf in the forwarders and the inputs.conf on the indexer.