I am looking at a similar problem. What I have observed is that when the heavy forwarder is unable to deliver to the upstream indexers ( Splunk Cloud in my case) due to network congestion, indexer unavailability, or network outage, then the heavy forwarder will refuse downstream connections, causing these errors. Once the buffer is full, it refuses any further connections. I think it is designed that way to force loadbalancing. Normally this is transient, as per your logs. In my case if the heavy forwarder goes into rejecting mode, it does not recover until it is restarted. ... View more

I had the same issue, after server patching broke the DB connector (jre path issue). Splunk was restarted by the root user, and all worked fine. Some time later, splunk was restarted and failed. A range of files were modified as owner:group root:root. I found this article - http://answers.splunk.com/answers/171236/fail-to-restart-splunk-after-installing-db-collect.html which described the issue and the fix ( chown of /opt/splunk back to splunk:splunk) ... View more