We have some files that we're monitoring through a universal forwarder and we're seeing behaviors where as the file is rotated some of the rotations aren't being indexed at all. So for each file we get everything in the file, or nothing at all.
We're seeing messages regularly that line up with the rotations that the file is being recognized as rotated, even when we've confirmed that the file is missed. The log sample below seems to be consistent both when the the file is indexed, and when it isn't.
05-06-2015 19:24:31.097 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/var/log/infra/task_log.log'.
05-06-2015 19:24:31.097 -0500 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/var/log/infra/task_log.log'.
05-06-2015 19:24:31.097 -0500 INFO BatchReader - Removed from queue file='/var/log/infra/task_log.log'.
These files have new content added to them at ~5minute intervals and rotate around once an hour, and we lose roughly 4 rotations worth of data a day. We also see regular messages lining with the extractions that we're exceeding the thruput maxKBps limit, which right now is at the default. From rough napkin math the total throughput for a day that we'd need is slightly above the maxKBps default.
What behavior should we be seeing with a file monitor when the maxKBps is exceeded? Should the forwards just hold their current seekptr until they've moved under some moving average? Or, as it seems to be working, are the current files finished in their entirety, and then a future reset of the file being dropped?
Also should we worry about the "Removed from queue" messages? They don't seem to be making a difference as we're seeing them for rotations that are being forwarded.
... View more