I am sending logs to a non-splunk server using syslog udp from the heavy forwarders which works fine. But recently the remote non-splunk server went down and the heavy forwarders were not able to reach it. As a result, there were multiple queues build-up which used up all of the resources to the point that all the existing log ingestion stopped on the heavy forwarders. Also, some of the heavy forwarders reported Splunk service not running.
Is there a way to prevent this from happening again in the future? What I want to make sure is if the remote server goes down in the future, the queues does not build up and the resources are not exhausted so that log ingestion still works?
