https://community.splunk.com/t5/Getting-Data-In/Forwarders-fail-to-make-connection-to-server/m-p/15863#M1838 <P>Frequently, our lightweight forwarders cannot connect to the Splunk server to send log tail output and we end up missing/losing those logs because they roll frequently. The client/forwarder will just log:</P> <PRE><CODE>06-21-2010 16:24:36.454 WARN TcpOutputProc - Failed to make a connection, will retry. 06-21-2010 16:24:56.495 INFO TcpOutputProc - Retrying connection to X.X.X.X:7080... 06-21-2010 16:24:56.496 WARN TcpOutputProc - Failed to make a connection, will retry. </CODE></PRE> <P>If I restart the splunk server, then it starts receiving data again. I've already set it up to restart twice a day, but that is not enough.</P> <P>The Splunk (4.1.3) server is Solaris 11 and a netstat -iv shows it has about 1-2 ierrs every 10 seconds, although the network folks say the switch port shows clean. I've attempted to tune TCP on the server, but it's made no difference.</P> <P>Also, when it stops receiving data, it will have a screen full of connections to localhost on the mgmt port shown in CLOSE_WAIT state similar to this:</P> <PRE><CODE>localhost.57325 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.57325 32768 0 32768 0 FIN_WAIT_2 localhost.33575 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.33575 32768 0 32768 0 FIN_WAIT_2 localhost.63480 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.63480 32768 0 32768 0 FIN_WAIT_2 </CODE></PRE> <P>Any ideas? Thanks!</P> Tue, 22 Jun 2010 04:15:12 GMT misschatter 2010-06-22T04:15:12Z https://community.splunk.com/t5/Getting-Data-In/Forwarders-fail-to-make-connection-to-server/m-p/15863#M1838 <P>Frequently, our lightweight forwarders cannot connect to the Splunk server to send log tail output and we end up missing/losing those logs because they roll frequently. The client/forwarder will just log:</P> <PRE><CODE>06-21-2010 16:24:36.454 WARN TcpOutputProc - Failed to make a connection, will retry. 06-21-2010 16:24:56.495 INFO TcpOutputProc - Retrying connection to X.X.X.X:7080... 06-21-2010 16:24:56.496 WARN TcpOutputProc - Failed to make a connection, will retry. </CODE></PRE> <P>If I restart the splunk server, then it starts receiving data again. I've already set it up to restart twice a day, but that is not enough.</P> <P>The Splunk (4.1.3) server is Solaris 11 and a netstat -iv shows it has about 1-2 ierrs every 10 seconds, although the network folks say the switch port shows clean. I've attempted to tune TCP on the server, but it's made no difference.</P> <P>Also, when it stops receiving data, it will have a screen full of connections to localhost on the mgmt port shown in CLOSE_WAIT state similar to this:</P> <PRE><CODE>localhost.57325 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.57325 32768 0 32768 0 FIN_WAIT_2 localhost.33575 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.33575 32768 0 32768 0 FIN_WAIT_2 localhost.63480 localhost.8089 32768 0 32768 0 CLOSE_WAIT localhost.8089 localhost.63480 32768 0 32768 0 FIN_WAIT_2 </CODE></PRE> <P>Any ideas? Thanks!</P> Tue, 22 Jun 2010 04:15:12 GMT https://community.splunk.com/t5/Getting-Data-In/Forwarders-fail-to-make-connection-to-server/m-p/15863#M1838 misschatter 2010-06-22T04:15:12Z https://community.splunk.com/t5/Getting-Data-In/Forwarders-fail-to-make-connection-to-server/m-p/15864#M1839 <P>Make sure you don't have an outputs.conf in your app directory ($SPLUNK_HOME/etc/apps/...) that has a tcpoutput to itself.</P> Thu, 23 Sep 2010 05:15:41 GMT https://community.splunk.com/t5/Getting-Data-In/Forwarders-fail-to-make-connection-to-server/m-p/15864#M1839 nate015 2010-09-23T05:15:41Z