Getting Data In

TcpOutputProc - The TCP output processor has paused the data flow

splunkcol
Builder

I open a new thread because in the previous one I was reviewing several errors at the same time

for this specific error message I have already read all the forum posts and the ones I have found on the internet but I am still having problems

• I have no Licence problems
• I don't have another output.conf file in the heavy forwarder
• I do not have a high performance or consumption in IOPS after consulting with the command iostats and iotop -

The devices through syslog send the logs to the heavy carrier that receives them and when reviewing them the logs are with the date and time updated all the time

For some reason the heavy forwarder doesn't constantly forward the logs to the indexers, as when querying with
index = * host = xxxx | statistics count per host _time

I see records but with delays of 8-10 hours.

When checking the logs in var / log / splunk / splunkd.log

I use grep xxxx splunkd.log to check the errors only from the host that interests me and that is where I see the message

09-21-2020 07:20:48.483 -0500 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=indexer inside output group default-autolb -group from host_src=xxxxx has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receivin g system's health in the Splunk Monitoring Console. It is probably not accepting data.

splunkcol_0-1600702525385.png

the heavy forwarder does not have high IOPS problems

The problem seems to be in the indexers that are not able to receive the information.

In that order of ideas, what do you recommend reviewing?

Labels (2)
0 Karma

thambisetty
SplunkTrust
SplunkTrust

The servers configured in outputs.conf are not performing well. there could be many reasons:

  1. network issue from Heavy forwarder to Indexer
  2. indexers are overwhelmed with events coming in or busy in serving requests from search head.

check all servers (indexers) in outputs.conf of Heavy forwarder are healthy (CPU and memory utilization).

check if you have deployed outputs.conf to indexers by mistake. generally indexers don't have outputs.conf. 

 

————————————
If this helps, give a like below.
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2026-2027 SplunkTrust is officially open. If ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...