Effects of syslog latency on a Unix application

[beaunewcomb]

We have a latency-sensitive application that must have latent-free logging output. The app is written to log out to a local file with no buffering. The app owner doesn't want to introduce any sort of buffer or queueing in the app to handle latency so logging doesn't slow down his process.

I'd like him to change his app to utilize syslog, and have syslog output to Splunk over the network. He's willing to re-rewrite his app as long as syslog handles all queueing of messages and doesn't slow down his app if there is latency in delivering the event. (ie, his app hands the event off to syslog, and moves on, leaving syslog to handle any latency)

Supporting documentation please

[dwaddle]

Even logging to a local file can introduce latency if a write(2) system call blocks for longer than intended. The only true way to have 100% latency free logging is not to log at all.

That said, typically the way the syslog(3) libc API works is with a local AF_UNIX datagram socket. This socket is (usually) located at /dev/log and is read by syslogd who then forwards the log messages over UDP to Splunk (or another syslog-speaking device).

I don't think you can say that writes to the AF_UNIX socket will never block under any circumstance. Kernel memory has to be set aside to store what has been written to but not read from that socket. Eventually, the writer has to block if no one is reading data out of it (that is unless the kernel will just throw a message away if the AF_UNIX socket is "full").

Some of these behaviors may be very OS/kernel specific. I would recommend discussing with your OS vendor how their libc implementation of syslog(3) handles these types of situations.

But, in normal operating modes, you might find logging via syslog(3) to have slightly less latency than writing to a file with write(2) simply because everything stays in memory from your process' perspective. Of course, this may depend on your kernel as well.

[beaunewcomb]

Thank you for the great response. Will use this going forward