topic Re: Q: BatchReader - Removed from queue WHY???? in Getting Data In

Q: BatchReader - Removed from queue WHY????

mkelderm — Mon, 28 Sep 2020 12:23:58 GMT

I see messages in the splunkd.log file:

09-07-2012 14:39:22.832 +0200 INFO BatchReader - Removed from queue file='/misc/tact/users/mk01232/agw/l1-iamprdagw25/reverse/MCS_RNL_2/http-reverse/extended/120727-A.log'.

Why is it removed from the queue? The file is asci, readable, correct permissions

Marc

Re: Q: BatchReader - Removed from queue WHY????

adamw — Sun, 09 Sep 2012 21:07:25 GMT

Marc,

Removed from queue typically means that Splunk has reached an EOF and is finished reading the file.

Thanks,
--adam

Re: Q: BatchReader - Removed from queue WHY????

yooperinmarylan — Tue, 24 Sep 2013 16:40:24 GMT

I am having the same problem with a log file (not syslog) that is continually written too. However Splunk (on a UF) decides its done reading and stops sending the log file. What do I need to do to correct this?

Re: Q: BatchReader - Removed from queue WHY????

bandit — Thu, 29 Jan 2015 15:25:58 GMT

I had the same issue for a csv log file with a header. My fix was to add the following line to my monitor rule in inputs.conf on the Universal Forwarder.

crcSalt= <SOURCE>

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty.