UDP is lossy, but can be tuned to a point to a "good enough" receipt rate. It is always best to use TCP if possible though.
In general you'll need to size the input buffers according to the input rate of messages. You have to pick a volume in bytes/second (as the kernel accounts memory use in bytes, rather than packets), this can be found out by measuring the actual traffic, or by checking the average message size and multiplying it with the desired message rate.
Once you have a number for your bytes/second rate, you'll have to pick a number how much time syslog-ng (or other syslog daemon) will spend on processing your messages (rather than receiving them). In normal cases, the round-trip-time between the reception of incoming messages is on the order of microseconds, sometimes a longer latency might happen. As described in one of the answers, this might be affected by the file system in use, or the processing applied to the incoming log stream.
I generally tend to plan for 3-5 seconds latencies, which means that syslog-ng will spend this time elsewhere, in the worst case.
Once these numbers are established we only need to make sure that our kernel based receive buffer can hold this amount of data.
In my example, let's say we work with 50k messages per second, 300 bytes on average and syslog-ng may suffer 5 second latencies. To offset this, we need a 75MB receive buffer. Today's servers dedicated to syslog processing should be able to allocate this amount of kernel memory to the syslog subsystem and remember, that this is the worst case and not a static allocation.
syslog-ng has a so-rcvbuf() option, but kernel limits also apply that prevent from increasing the receive buffer above a certain threshold. To increase the threshold you'll need to change 'sys.net.core.rmem_max' value:
# echo 75000000 > /proc/sys/net/core/rmem_max
And then use this option for your udp receiver in syslog-ng.conf:
source s_net {
udp(so-rcvbuf(75000000));
};
If you want to be absolutely sure, you can check syslog-ng with strace to see if it really increased the socket buffer.
In general version 3.3 of syslog-ng (released early 2011, currently at 3.3.6), should have no issue processing 100k messages per second from a single source. When processing messages from a large number of clients, this number can also be scaled to 1M msg/sec.
That should be more than anyone needs, even though certainly this number depends on the configuration used for syslog-ng.
... View more