How can I increase the universal forwarder buffer size to insure logs are not lost when site becomes isolated?

[nickcc]

The network connectivity at one of my sites drops on occasions. So, I was reading best practice is to configure a syslog collector to ingest logs via UDP 514 at local site and forward those to an indexer offsite via universal forwarder TCP 9997.

Now, if the site loses network connectivity and the collector no longer has the capability to forward logs to the indexer, will the universal forwarder (on collector) buffer the logs to make certain nothing is missed once everything comes up and its able to send logs to the indexer? I saw that the use persistent queues are not available for specific types and splunktcp (input from Splunk forwarder) was on the list.

My question is, how can I increase the buffer size (if all) to insure my logs are not lost when my site becomes isolated. Also if I have this completely wrong and there is a better way, please let me know. Any assistance would be greatly appreciated

[nickhills · Accepted solution]

The recommended approach is to send Syslog messages to a syslog receiver like syslog-ng.
The syslog receiver will write the events to file, and then you use a forwarder to collect the results from file, and send to your indexers.

This breaks the "real-time" dependency for syslog, and means that you can tolerate network disruption between the UF-indexers, rather than risk loosing syslog messages in flight.