Getting Data In

Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group {{ redacted }} has been blocked for 30 seconds

sochase
Observer

I have a new Splunk deployment with a multi-site index cluster. I currently have setup heavy forwarders using indexer discovery and assigning them to the primary site. In my DMC all health checks and index cluster status look good, and we as the index cluster status when looking on the master. In splunkd.log on the index peers and master, I have no errors. I have setup an ssl input on the index cluster and do not have a non-ssl input enabled. I have configured the heavy forwarders output.conf to useSSL. To keep things simple right now, I am not requiring a client cert in the indexer's input.conf.

The problem I am seeing is in the heavy forwarder's splunkd.log, and it states: Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group {{ redacted }} has been blocked for 30 seconds

I have verified connectivity to the master and index peers from the heavy forwarders and have verified connectivity to the input port on the index peers from the heavy forwarders.

Any thoughts?

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @kipkip,

This error shows that HF cannot send data to indexers. You didn't mention which instance you are running Distributed Monitoring Console. You should check the status of indexers on the Monitoring console. There may be problems with Indexers (disk space, not running, etc.) or communication between HF and indexers.

If this reply helps you an upvote is appreciated.
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you followed steps given here , if yes then I'll suggest you to provide outputs.conf from your Heavy Forwarder and inputs.conf from your Indexer (Mask any sensitive data).

0 Karma

kipkip
Loves-to-Learn

@harsmarvania57  I am receving that exact message on my Splunk Heavy forwarder.  Here is the breakdown of my environment:

1. Splunk Deployer

3. Search Head Cluster

3 Splunk Indexers

1. Master Cluster

1 Deployment/Licence server

I notice data stopped coming in about 5 days ago. However, I am receiving this message on the HF:

TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest= inside output group default-autolb-group from host_src=heavy-forwarder.example.com has been blocked for blocked_seconds=1440. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Receiving port : 9997 is enabled on Splunk HF Dashboard but port 9997 is NOT LISTEN on the HF command line

I would appreciate any help to resolve this issue as soon as possible.

 

 

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...