The TCP output processor has paused the data flow.

[Rushilgupta02]

After my system gets patched, connection from host (nginx servers) to splunk gets cut (sometimes). This causes logs to not get populated on splunk. For example- I have 5 nginx servers, all of them get patched but 3 of them loose connection and this is random. I have pasted my logs down below, any guidance on how to fix this issue?

Logs-
11-02-2025 03:19:19.345 +0000 INFO AutoLoadBalancedConnectionStrategy [3292 TcpOutEloop] - Connected to idx=1x.xxx.x.x:9997:3, pset=0, reuse=0. autoBatch=1
11-02-2025 03:19:49.245 +0000 INFO AutoLoadBalancedConnectionStrategy [3292 TcpOutEloop] - Connected to idx=1x.xxx.x.x:9997:3, pset=0, reuse=0. autoBatch=1
11-02-2025 03:20:00.697 +0000 INFO DC:DeploymentClient [3141 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-02-2025 03:20:07.945 +0000 WARN TcpOutputProc [3289 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=proxy.splunk.local inside output group nginx from host_src=us-ng3 has been blocked for blocked_seconds=18400. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

[isoutamo]

What you actually mean with this "connection from host (nginx servers) to splunk gets cut (sometimes)."?
Is the connection always down, will it start to working after some time or after something has done? Or something else?

[PrewinThomas]

@Rushilgupta02 

Any Firewall/SELinux reset happened after patching? Did you restart UF after patching? Sometimes UF service may not restart cleanly during patching. Also verify DNS resolution for proxy.splunk.local

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[Rushilgupta02]

adding to this, all my ports are open, firewall is fine.....there should be no changes other than the ec2 instance rebooting.

[gcusello]

Hi @Rushilgupta02 ,

did you checked the local firewalls on the nginx servers?

Ciao.

Giuseppe