Applying quarantine and removing quarantine

[bhsakarchourasi]

Hi All,

This is kind of similar issue as mention on below link but since it was unanswered posting it again.
https://answers.splunk.com/answers/211112/applying-quarantine-removing-quarantine.html

We have installed Universal forwarder on a new server to send logs to Splunk cloud, since we didn't have direct connectivity to indexers we are sending logs to heavy forwarder and we didn't have connectivity to DS as well so we are doing manual configuration in /etc/system/local, but we are getting below errors in UF.

11-01-2019 16:37:21.594 +0800 INFO  TcpOutputProc - Removing quarantine from idx=xx.xx.xx.xx:9997
11-01-2019 16:37:21.780 +0800 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
11-01-2019 16:37:21.966 +0800 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
11-01-2019 16:37:21.967 +0800 WARN  TcpOutputProc - Applying quarantine to ip=xx.xx.xx.xx port=9997 _numberOfFailures=2
11-01-2019 16:37:22.142 +0800 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-01-2019 16:37:25.062 +0800 WARN  TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group splunk has been blocked for 300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

We have some other universal forwarders sending logs to the same heavy forwarder and it's working fine.

Thanks in advance any help will be much appreciated.

Thanks.

[codebuilder]

The log entries you posted are expected behavior in a number of situations.
The top two are:
1. Incorrect pass4Symkey value between forwarder and cluster [general], or indexer master using indexer discovery.
2. Indexer cluster using indexer discover was cycled (rolling restart), and the forwarder has not received an updated list of available indexers. The timeout for discovery is configurable.