Getting Data In

Forwarder tcpout_connections blocked

Chris_R_
Splunk Employee
Splunk Employee

This configuration is two 3.4.2 forwarders -> two 4.1.2 indexers.
Forwarders have two UDP inputs & two seperate assigned sourcetypes on these UDP inputs, props/transforms/outputs entries are doing _TCP_ROUTING to two seperate indexers.
Config seems ok for the most part. However they are getting constantly blocked tcpout_connections messages in metrics.log

splunkd.log Error on the forwarders

07-07-2010 06:11:29.452 WARN  TcpOutputProc - TcpSendThread: Connection to server lost - retrying: Broken pipe  
07-07-2010 06:11:29.452 WARN  TcpOutputProc - Connection dropped by Indexer. Possible version mismatch with indexer. Please check compatibility with indexer version  

splunkd.log errors on the indexer

07-08-2010 01:15:13.501 ERROR TcpInputProc - Error encountered for connection from host=< ip address >, ip=< ip address >. Timeout  
07-08-2010 01:15:13.501 INFO  TcpInputProc - Hostname=< ip address > closed connection  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel source::udp:515|host::192.168.88.25|somesourcetypel|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:514|host::192.168.8.204|somesourcetypee|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.26|somesourcetype|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::/opt/splunk/var/log/splunk/splunklogger.log|host::NCCForwarder|splunklogger|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.27|somesourcetype|remoteport::41108" ended without a done-key  

one other odd entry i see in the inputs.conf of the indexers, seems like this is a older spec file setting to route certain data to queues instead of letting splunk do it automatically?

[splunktcp]
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;
Tags (2)
0 Karma

Chris_R_
Splunk Employee
Splunk Employee

GK: These are full forwarders, here's the outputs from a forwarder

[tcpout]  
indexAndForward = false  


[tcpout:stonegateGroup]
disabled = false  
server=10.20.12.35:9001  

[tcpout:fortimailGroup]  
disabled = false  
server=10.20.12.33:9997  

and the inputs.conf from a indexer

[default]  
index = default  
host = fortimailsplunk  
_rcvbuf = 196608  

[monitor://$SPLUNK_HOME/var/spool/splunk]  
move_policy = sinkhole  

[fschange:$SPLUNK_HOME/etc]  
signedaudit = true  
sendEventMaxSize = -1  
recurse = true  
pollPeriod = 600  
filesPerDelay = 10  
delayInMills = 100  
followLinks = false  
fullEvent = false  
hashMaxSize = -1  

[splunktcp]  
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;  

Note: I had them remove the tcp route = stanza seems to not be blocking this morning, could be a slower day...but i'll know for sure next week

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

That route is in fact in the etc/system/default/inputs.conf for 4.x machines. Someone might have copied it over. Don't mess with it.

Please clarify if these are heavy forwarders, or LWF's tweaked to collect UDP as well? It would be helpful to see the outputs.conf in the forwarders and the inputs.conf on the indexer.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...