Getting Data In

WatchedFile - Will begin reading at offset

echalex
Builder

Hi,
I have a weird issue with a bunch of files that I have recently started indexing. A whole bunch of these will end up in the index with missing events (typically missing 1 or 2 events).

For these files I'm seeing reports like:

09-27-2012 11:08:05.908 +0300 INFO  WatchedFile - Will begin reading at offset=253 for file='/path/to/file.log'

Why does the forwarder skip events like this?

0 Karma
1 Solution

matthew_lawrenc
Explorer

We've observed the same behaviour, in our case the cause was the tailing process does duplication detection with a checksum on the first and last 256 bytes of a file (so as to not index the same file twice).

We had two files with identical content being monitored.

Our resolution was to use crcSalt = A and crcSalt = B in each stanza which ensured they had differing checksums (we could do this because the duplicate files were listed in two different monitor stanzas).

You should also beware of changing the crcSalt on a running system as it will cause previously indexed files to be re-indexed.

Over a 10 minute window our splunkd.log looked like;

INFO  WatchedFile - Will begin reading at offset=1575 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=1890 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2115 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2205 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2340 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2700 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2970 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=5490 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=6795 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=7560 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=9810 for file='/tmp/logtset/test_2013101419.log'.

View solution in original post

matthew_lawrenc
Explorer

We've observed the same behaviour, in our case the cause was the tailing process does duplication detection with a checksum on the first and last 256 bytes of a file (so as to not index the same file twice).

We had two files with identical content being monitored.

Our resolution was to use crcSalt = A and crcSalt = B in each stanza which ensured they had differing checksums (we could do this because the duplicate files were listed in two different monitor stanzas).

You should also beware of changing the crcSalt on a running system as it will cause previously indexed files to be re-indexed.

Over a 10 minute window our splunkd.log looked like;

INFO  WatchedFile - Will begin reading at offset=1575 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=1890 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2115 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2205 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2340 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2700 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2970 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=5490 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=6795 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=7560 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=9810 for file='/tmp/logtset/test_2013101419.log'.

markbarber21
Path Finder

FYI - "Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed."

From: http://docs.splunk.com/Documentation/Splunk/6.6.3/Data/Howlogfilerotationishandled

0 Karma

_gkollias
SplunkTrust
SplunkTrust

Thanks for this. I experiencing this as well, however my log files are not completely the same...I have something like the following:

WatchedFile - Will begin reading at offset= for file='/log/path/1/logs/file.log

WatchedFile - Will begin reading at offset= for file='/log/path/2/logs/file.log

The log paths are slightly different, but the log file names are the same. I am also already using crcSalt, but this just began 3 days ago. Still no solution found

0 Karma

BenAveling
Path Finder

I assume the log files are rotating? Do you know where in your log files the missing messages would have been?

0 Karma

eashwar
Communicator

hey brother i got some same issue now for the past 2 days. the whole app i developed is not working. please let me know if you know the answer

0 Karma

yannK
Splunk Employee
Splunk Employee

Are you using the option ?


followTail=true

this option is misleading and for rotating files cause the offset to not start at 0, see http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

Please remove it.

echalex
Builder

Nope, I'm not using followTail=true, because I know - by experience - that it's not a good idea.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...