Getting Data In

Effects of syslog latency on a Unix application

beaunewcomb
Communicator

We have a latency-sensitive application that must have latent-free logging output. The app is written to log out to a local file with no buffering. The app owner doesn't want to introduce any sort of buffer or queueing in the app to handle latency so logging doesn't slow down his process.

I'd like him to change his app to utilize syslog, and have syslog output to Splunk over the network. He's willing to re-rewrite his app as long as syslog handles all queueing of messages and doesn't slow down his app if there is latency in delivering the event. (ie, his app hands the event off to syslog, and moves on, leaving syslog to handle any latency)

Supporting documentation please

Tags (3)

dwaddle
SplunkTrust
SplunkTrust

Even logging to a local file can introduce latency if a write(2) system call blocks for longer than intended. The only true way to have 100% latency free logging is not to log at all.

That said, typically the way the syslog(3) libc API works is with a local AF_UNIX datagram socket. This socket is (usually) located at /dev/log and is read by syslogd who then forwards the log messages over UDP to Splunk (or another syslog-speaking device).

I don't think you can say that writes to the AF_UNIX socket will never block under any circumstance. Kernel memory has to be set aside to store what has been written to but not read from that socket. Eventually, the writer has to block if no one is reading data out of it (that is unless the kernel will just throw a message away if the AF_UNIX socket is "full").

Some of these behaviors may be very OS/kernel specific. I would recommend discussing with your OS vendor how their libc implementation of syslog(3) handles these types of situations.

But, in normal operating modes, you might find logging via syslog(3) to have slightly less latency than writing to a file with write(2) simply because everything stays in memory from your process' perspective. Of course, this may depend on your kernel as well.

beaunewcomb
Communicator

Thank you for the great response. Will use this going forward

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...