The TCP output processor has paused the data flow.

[Rushilgupta02]

After my system gets patched, connection from host (nginx servers) to splunk gets cut (sometimes). This causes logs to not get populated on splunk. For example- I have 5 nginx servers, all of them get patched but 3 of them loose connection and this is random. I have pasted my logs down below, any guidance on how to fix this issue?

Logs-
11-02-2025 03:19:19.345 +0000 INFO AutoLoadBalancedConnectionStrategy [3292 TcpOutEloop] - Connected to idx=1x.xxx.x.x:9997:3, pset=0, reuse=0. autoBatch=1
11-02-2025 03:19:49.245 +0000 INFO AutoLoadBalancedConnectionStrategy [3292 TcpOutEloop] - Connected to idx=1x.xxx.x.x:9997:3, pset=0, reuse=0. autoBatch=1
11-02-2025 03:20:00.697 +0000 INFO DC:DeploymentClient [3141 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-02-2025 03:20:07.945 +0000 WARN TcpOutputProc [3289 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=proxy.splunk.local inside output group nginx from host_src=us-ng3 has been blocked for blocked_seconds=18400. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

[burwell]

Hi. What version of Splunk are you running?

I ran into a bad bug with both Splunk Enterprise 9.3.7 and 9.4.5. The heavy forwarders sending to DNS load balanced indexers get TCPOUT blocked.  This bug does not appear to be on the known issues despite many attempts by me trying to get it added there. It does not happen with 9.2.4.

The Splunk JIRA that was opened is SPL-288904 

The bug is said to be fixed in the upcoming releases 9.3.9 , 9.4.7, 10.0.3, 10.1
Hopefully soon.

A workaround is the setting of dnsResolutionInterval in outputs.conf

dnsResolutionInterval = <integer>
* The base time interval, in seconds, at which indexer Domain Name Server
  (DNS) names are resolved to IP addresses.
* This is used to compute runtime dnsResolutionInterval as follows:
  Runtime interval =
   'dnsResolutionInterval' + (number of indexers in server settings - 1) * 30.
* The DNS resolution interval is extended by 30 seconds for each additional
  indexer in the server setting.
* Default: 300 seconds (5 minutes)

Splunk had recommended we set dnsResolutionInterval =480 (tcpout blocked). I tried 1000 (also blocked). 
I have set it to 10000 (ie 10,000) and after ~ 3 days this seems to be working.

[isoutamo]

What you actually mean with this "connection from host (nginx servers) to splunk gets cut (sometimes)."?
Is the connection always down, will it start to working after some time or after something has done? Or something else?

[PrewinThomas]

@Rushilgupta02 

Any Firewall/SELinux reset happened after patching? Did you restart UF after patching? Sometimes UF service may not restart cleanly during patching. Also verify DNS resolution for proxy.splunk.local

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[Rushilgupta02]

adding to this, all my ports are open, firewall is fine.....there should be no changes other than the ec2 instance rebooting.

[gcusello]

Hi @Rushilgupta02 ,

did you checked the local firewalls on the nginx servers?

Ciao.

Giuseppe