Getting Data In

TcpOutputProc - The TCP output processor has paused the data flow

splunkcol
Builder

I open a new thread because in the previous one I was reviewing several errors at the same time

for this specific error message I have already read all the forum posts and the ones I have found on the internet but I am still having problems

• I have no Licence problems
• I don't have another output.conf file in the heavy forwarder
• I do not have a high performance or consumption in IOPS after consulting with the command iostats and iotop -

The devices through syslog send the logs to the heavy carrier that receives them and when reviewing them the logs are with the date and time updated all the time

For some reason the heavy forwarder doesn't constantly forward the logs to the indexers, as when querying with
index = * host = xxxx | statistics count per host _time

I see records but with delays of 8-10 hours.

When checking the logs in var / log / splunk / splunkd.log

I use grep xxxx splunkd.log to check the errors only from the host that interests me and that is where I see the message

09-21-2020 07:20:48.483 -0500 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=indexer inside output group default-autolb -group from host_src=xxxxx has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receivin g system's health in the Splunk Monitoring Console. It is probably not accepting data.

splunkcol_0-1600702525385.png

the heavy forwarder does not have high IOPS problems

The problem seems to be in the indexers that are not able to receive the information.

In that order of ideas, what do you recommend reviewing?

Labels (2)
0 Karma

thambisetty
SplunkTrust
SplunkTrust

The servers configured in outputs.conf are not performing well. there could be many reasons:

  1. network issue from Heavy forwarder to Indexer
  2. indexers are overwhelmed with events coming in or busy in serving requests from search head.

check all servers (indexers) in outputs.conf of Heavy forwarder are healthy (CPU and memory utilization).

check if you have deployed outputs.conf to indexers by mistake. generally indexers don't have outputs.conf. 

 

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...