topic WatchedFile - Will begin reading at offset in Getting Data In

WatchedFile - Will begin reading at offset

echalex — Thu, 27 Sep 2012 09:24:54 GMT

Hi,
I have a weird issue with a bunch of files that I have recently started indexing. A whole bunch of these will end up in the index with missing events (typically missing 1 or 2 events).

For these files I'm seeing reports like:

09-27-2012 11:08:05.908 +0300 INFO  WatchedFile - Will begin reading at offset=253 for file='/path/to/file.log'

Why does the forwarder skip events like this?

Re: WatchedFile - Will begin reading at offset

yannK — Thu, 27 Sep 2012 15:36:39 GMT

Are you using the option ?



followTail=true

this option is misleading and for rotating files cause the offset to not start at 0, see http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

Please remove it.

Re: WatchedFile - Will begin reading at offset

echalex — Fri, 28 Sep 2012 06:11:59 GMT

Nope, I'm not using followTail=true, because I know - by experience - that it's not a good idea.

Re: WatchedFile - Will begin reading at offset

eashwar — Fri, 10 May 2013 08:43:05 GMT

hey brother i got some same issue now for the past 2 days. the whole app i developed is not working. please let me know if you know the answer

Re: WatchedFile - Will begin reading at offset

BenAveling — Fri, 06 Sep 2013 07:12:42 GMT

I assume the log files are rotating? Do you know where in your log files the missing messages would have been?

Re: WatchedFile - Will begin reading at offset

matthew_lawrenc — Tue, 15 Oct 2013 20:40:52 GMT

We've observed the same behaviour, in our case the cause was the tailing process does duplication detection with a checksum on the first and last 256 bytes of a file (so as to not index the same file twice).

We had two files with identical content being monitored.

Our resolution was to use crcSalt = A and crcSalt = B in each stanza which ensured they had differing checksums (we could do this because the duplicate files were listed in two different monitor stanzas).

You should also beware of changing the crcSalt on a running system as it will cause previously indexed files to be re-indexed.

Over a 10 minute window our splunkd.log looked like;

INFO  WatchedFile - Will begin reading at offset=1575 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=1890 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2115 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2205 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2340 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2700 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2970 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=5490 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=6795 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=7560 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=9810 for file='/tmp/logtset/test_2013101419.log'.

Re: WatchedFile - Will begin reading at offset

_gkollias — Fri, 08 Aug 2014 17:06:51 GMT

Thanks for this. I experiencing this as well, however my log files are not completely the same...I have something like the following:

WatchedFile - Will begin reading at offset= for file='/log/path/1/logs/file.log

WatchedFile - Will begin reading at offset= for file='/log/path/2/logs/file.log

The log paths are slightly different, but the log file names are the same. I am also already using crcSalt, but this just began 3 days ago. Still no solution found

Re: WatchedFile - Will begin reading at offset

markbarber21 — Tue, 31 Jul 2018 19:50:01 GMT

FYI - "Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed."

From: http://docs.splunk.com/Documentation/Splunk/6.6.3/Data/Howlogfilerotationishandled