I see messages in the splunkd.log file:
09-07-2012 14:39:22.832 +0200 INFO BatchReader - Removed from queue file='/misc/tact/users/mk01232/agw/l1-iamprdagw25/reverse/MCSRNL2/http-reverse/extended/120727-A.log'.
Why is it removed from the queue? The file is asci, readable, correct permissions
I had the same issue for a csv log file with a header. My fix was to add the following line to my monitor rule in inputs.conf on the Universal Forwarder.
crcSalt = <string> * Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.) * If set, <string> is added to the CRC. * If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC. When crcSalt is invoked, it is usually set to <SOURCE>. * Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. * Defaults to empty.
I am having the same problem with a log file (not syslog) that is continually written too. However Splunk (on a UF) decides its done reading and stops sending the log file. What do I need to do to correct this?