UDP droppage because of ext4 filesystem:

[balbano]

I have been having an issue where one of my 2 log servers have dropping a tremendous amount of UDP packet data (from syslog-ng/rsyslog based traffic).

One of 2 log servers has been dropping UDP packets like crazy. However the other one was fine.

While the changes that were mentioned in here did improve the situation, the drop rate was at a significant level where it was ridiculous.

After banging my head over it for why one was dropping and the other one wasnt dropping, I realized a key difference in the log servers: The working server was having the logs write to an ext3 partition and the server dropping logs was writing to an ext4 filesystem.

As a test, I moved the log destination to an ext3 filesystem with default settings and now its working fine.

Now the question, what are the appropriate ext4 settings for receiving syslog-ng / rsyslog data?

This is what I currently have setup (which is causing the UDP Droppage):

/dev/$my_device /$my_log_dest ext4 noatime,data=writeback,defaults,acl 1 2

I suspect its possibly my journaling option "data=writeback" but I'm not for certain.

Can someone give some insight on this?

Thanks.

Brian

[Wilcooley]

Sorry for bringing up an old question but I happened upon this after some recent IRC discussion.

I am curious about the size of the journal in your ext4 file system. With ext3 (and presumably ext4 by extension), having too small of a journal was a source of stalls or hangs when writing. This could happen if you initially created a small file system and then grew it significantly. You can find out with the dumpe2fs command (sub '4' for '2' if on EL5):

dumpe2fs -h /dev/XXX |grep Journal

It would also be interesting to know what features are enabled; you can get that with either the dumpe2fs or tune2fs -l command.

Also, what kernel version & distro are you using?

I am assuming that you're using Splunk as the UDP listener and not feeding via an intermediary syslog server? (My Splunk never sees UDP traffic because I feed it via rsyslog.)