Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group {{ redacted }} has been blocked for 30 seconds

[sochase]

I have a new Splunk deployment with a multi-site index cluster. I currently have setup heavy forwarders using indexer discovery and assigning them to the primary site. In my DMC all health checks and index cluster status look good, and we as the index cluster status when looking on the master. In splunkd.log on the index peers and master, I have no errors. I have setup an ssl input on the index cluster and do not have a non-ssl input enabled. I have configured the heavy forwarders output.conf to useSSL. To keep things simple right now, I am not requiring a client cert in the indexer's input.conf.

The problem I am seeing is in the heavy forwarder's splunkd.log, and it states: Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group {{ redacted }} has been blocked for 30 seconds

I have verified connectivity to the master and index peers from the heavy forwarders and have verified connectivity to the input port on the index peers from the heavy forwarders.

Any thoughts?

[scelikok]

Hi @kipkip,

This error shows that HF cannot send data to indexers. You didn't mention which instance you are running Distributed Monitoring Console. You should check the status of indexers on the Monitoring console. There may be problems with Indexers (disk space, not running, etc.) or communication between HF and indexers.

[harsmarvania57]

Hi,

Have you followed steps given here , if yes then I'll suggest you to provide outputs.conf from your Heavy Forwarder and inputs.conf from your Indexer (Mask any sensitive data).

[kipkip]

@harsmarvania57  I am receving that exact message on my Splunk Heavy forwarder.  Here is the breakdown of my environment:

1. Splunk Deployer

3. Search Head Cluster

3 Splunk Indexers

1. Master Cluster

1 Deployment/Licence server

I notice data stopped coming in about 5 days ago. However, I am receiving this message on the HF:

TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest= inside output group default-autolb-group from host_src=heavy-forwarder.example.com has been blocked for blocked_seconds=1440. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Receiving port : 9997 is enabled on Splunk HF Dashboard but port 9997 is NOT LISTEN on the HF command line

I would appreciate any help to resolve this issue as soon as possible.