https://community.splunk.com/t5/Getting-Data-In/Why-am-I-getting-timeout-issues-from-Splunk-forwarders-to-the/m-p/229519#M44663 <P>I have a random time out issue from Splunk forwarders to the Splunk intermediate (heavy) forwarder. </P> <P>When I do <CODE>netstat -al | grep 9997</CODE>, I get:</P> <PRE><CODE>splunkndx-9997 SYN_SENT splunkndx-9997 FIN_WAIT1 </CODE></PRE> <P>from universal forwarder side splunkd.log </P> <PRE><CODE>08-16-2016 02:08:50.047 +0000 WARN TcpOutputProc - Forwarding to indexer group sid_9997 blocked for 1600 seconds. 08-16-2016 02:08:50.978 +0000 WARN TcpOutputProc - Cooked connection to ip=*.*.*.*:9997 timed out 08-16-2016 02:09:11.981 +0000 WARN TcpOutputProc - Cooked connection to ip=*.*.*.*:9997 timed out </CODE></PRE> <P>When I do Telnet, 9 times out of 10, I get a timeout. Is there any config I can tune up or the heavy forwarders need to be resized? The CPU usage is less than 10%. Any suggestions?</P> Mon, 15 Aug 2016 22:26:36 GMT raindrop18 2016-08-15T22:26:36Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-getting-timeout-issues-from-Splunk-forwarders-to-the/m-p/229519#M44663 <P>I have a random time out issue from Splunk forwarders to the Splunk intermediate (heavy) forwarder. </P> <P>When I do <CODE>netstat -al | grep 9997</CODE>, I get:</P> <PRE><CODE>splunkndx-9997 SYN_SENT splunkndx-9997 FIN_WAIT1 </CODE></PRE> <P>from universal forwarder side splunkd.log </P> <PRE><CODE>08-16-2016 02:08:50.047 +0000 WARN TcpOutputProc - Forwarding to indexer group sid_9997 blocked for 1600 seconds. 08-16-2016 02:08:50.978 +0000 WARN TcpOutputProc - Cooked connection to ip=*.*.*.*:9997 timed out 08-16-2016 02:09:11.981 +0000 WARN TcpOutputProc - Cooked connection to ip=*.*.*.*:9997 timed out </CODE></PRE> <P>When I do Telnet, 9 times out of 10, I get a timeout. Is there any config I can tune up or the heavy forwarders need to be resized? The CPU usage is less than 10%. Any suggestions?</P> Mon, 15 Aug 2016 22:26:36 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-getting-timeout-issues-from-Splunk-forwarders-to-the/m-p/229519#M44663 raindrop18 2016-08-15T22:26:36Z https://community.splunk.com/t5/Getting-Data-In/Why-am-I-getting-timeout-issues-from-Splunk-forwarders-to-the/m-p/229520#M44664 <P>This seems to be problem with routing taken to reach the forwarder rather than Splunk or CPU usage. Can you verify entries in routing table? </P> Tue, 16 Aug 2016 05:23:52 GMT https://community.splunk.com/t5/Getting-Data-In/Why-am-I-getting-timeout-issues-from-Splunk-forwarders-to-the/m-p/229520#M44664 hardikJsheth 2016-08-16T05:23:52Z