ERROR TcpInputProc - Indexer not receiving data from forwarder

cdubs — Tue, 29 Sep 2020 18:40:29 GMT

Hi all, I am getting these errors in my log files. First is from the spunkd.log from the indexer and second is is from the splunkd.log on the forwarder. I have done multiple searches on Splunk answers, but I haven't found one that pertain to both. It obvious in the error log on the forwarder that the connection is refused however I can telnet to the port 9997. What am I missing? This was all working until upgrading to 7.02. Thankfully this is just a test machine and not in production. Please let me know what I can provide you all to assist me in troubleshooting such as .conf/log files etc. I will continue to search & troubleshoot, but at this point I am loss.

Splunk IDX Error:

ERROR TcpInputProc - Message rejected. Received unexpected message of size=369295616 bytes from src=xxx.xx.xxx.xx:64529 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

ERROR TcpInputProc - Message rejected. Received unexpected message of size=369295616 bytes from src=xxx.xx.xxx.xx:61330 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

Splunk UF Error:

WARN TcpOutputProc - Applying quarantine to ip=xxx.xx.xxx.xx port=9997 _numberOfFailures=2
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 3601 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 3701 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xx.xxx.xx_8089_ _XA5D5CF2-F5DB-4F1F-BAE9-909B3A7FEA00

WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 3801 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xx.xxx.xx_8089_XA5D5CF2-F5DB-4F1F-BAE9-909B3A7FEA00
INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xx.xxx.xx_8089XA5D5CF2-F5DB-4F1F-BAE9-909B3A7FEA00
INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xx.xxx.xx_8089_XA5D5CF2-F5DB-4F1F-BAE9-909B3A7FEA00
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 3901 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

INFO TcpOutputProc - Removing quarantine from idx=xxx.xx.xxx.xx:9997

ERROR TcpOutputFd - Connection to host=xxx.xx.xxx:9997 failed
ERROR TcpOutputFd - Connection to host=xxx.xx.xxx:9997 failed

Thank You

Re: ERROR TcpInputProc - Indexer not receiving data from forwarder

Kendrick821 — Mon, 26 Mar 2018 03:36:18 GMT

Hi, will you be able to post the inputs.conf of indexer and outputs.conf of UF?

topic ERROR TcpInputProc - Indexer not receiving data from forwarder in Getting Data In

ERROR TcpInputProc - Indexer not receiving data from forwarder

Re: ERROR TcpInputProc - Indexer not receiving data from forwarder