We're getting bunch of these exceptions on our Universal Forwarders...any help would be appreciated and I can provide more info if needed...
1) ERROR TailReader -File will not be read, seekptr checksum did not match
it says file will not be read. Does that mean it's ignoring the live log which is logged in the path specified..?
ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/jr_LCMI/log/server.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info
2) INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file
Does that mean it's re-indexing entire file again..?
INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/ws/ATG-Data/home/servers/ku_JVM00/logs/apps.log'
INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/ws/server/ra_JVM00/log/server.log'.
As soon as I restart UFs, I see the message below. What do the offset numbers mean..?
INFO WatchedFile - Will begin reading at offset=0 for file='file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log'.
INFO WatchedFile - Will begin reading at offset=90 for file='/opt/app/ws/server/ra_JVM00/log/server.log'.
INFO WatchedFile - Will begin reading at offset=180 for file='file=/opt/app/ws/server/kv_JVM00/log/responseTime.2016-05-04.log'.
Configs on my universal forwarders:
inputs.conf
[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
index=testenv
[monitor:///opt/app/ws/server/*/log/responseTime.*.log]
Sourcetype=responseTime
index=testenv
[monitor:///opt/app/ws/ATG-Data/home/servers/*/logs/apps.log]
Sourcetype=apps
index=testenv
outputs.conf
#conpressed and useACK were not set for some of the UFs
#splhfserver is a HF which route the data to Splunk, it's not indexing locally..
[tcpout]
defaultGroup = splhfdataforwarder
compressed = true
useACK = true
[tcpout:splhfdatafowarder]
Server=splhfserver:9997
We ended up doing something like -
[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
crcSalt = <source>
initCrcLength = 2000
index=testenv
Here <source>means have you typed exact path of the file or just <source> as is?
when i add the above settings, i still see this messages on splunkd.logs, how did you resolve this...?
File too small to check seekcrc, probably truncated. Will re-read entire file=...filepath
File too small to check seekcrc, probably truncated. Will re-read entire file=...filepath
05-05-2016 16:09:54.601 -0500 INFO WatchedFile - Logfile truncated while open, original pathname file='/opt/app/ws/server/ra_JVM00/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.602 -0500 INFO WatchedFile - Logfile truncated while open, original pathname ffile='/opt/app/ws/server/ra_JVM04/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.605 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/ws/server/ra_JVM02/log/server.log'.
So how big is this file? - /opt/app/ws/server/ra_JVM02/log/server.log
this is what i see from the path /opt/app/ws/server/ra_JVM02/log/server.log
$ cat server.log | wc -l
1205
$ cat server.log | wc -c
236896
Adding a crcSalt helped me to get rid of this messages...
ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
Adding a initCrcLength = 2000 OR followTail =1, doesn't help me with this issue...not sure how i can verify if this message is a sign of re-indexing or not..?
File too small to check seekcrc, probably truncated. Will re-read entire file=...filepath
File too small to check seekcrc, probably truncated. Will re-read entire file=...filepath
05-05-2016 16:09:54.601 -0500 INFO WatchedFile - Logfile truncated while open, original pathname file='/opt/app/ws/server/ra_JVM00/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.602 -0500 INFO WatchedFile - Logfile truncated while open, original pathname ffile='/opt/app/ws/server/ra_JVM04/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.605 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/ws/server/ra_JVM02/log/server.log'.
Will go with the above said option, is there any ways to check if logs are re-indexed or double indexed...?
I did follow this post, but the numbers do not match...
ERROR TailingProcessor - Ignoring path
speaks about it.
ddrilic,
If i use a crcSalt on all there sources, is there any chance of re-indexing the data. The answers seems to be relevant to v4.1.5 of splunk.
it's applicable to 6.4 - 6.4 - Inputsconf
-- If i use a crcSalt on all there sources, is there any chance of re-indexing the data.
Depends, I guess, on the value of initCrcLength. If it's, let's say, 2000, instead of the default 256, you probably should be ok.
Actually i'm having hard time in figuring it out....
The actual path of the monitor stanza would include..
[monitor:///opt/app/ws/server/*/log/server.log]
/opt/app/ws/server/ra_JVM00/log/server.log
/opt/app/ws/server/ra_JVM01/log/server.log
/opt/app/ws/server/pr_INS00/log/server.log
/opt/app/ws/server/pr_INS02/log/server.log
Can you suggest me which would the best option to go with...
[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
crcSalt = <SOURCE>
index=testenv
OR
[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
initCrcLength = 2000
index=testenv
Option 2, avoids chances of duplication.
somesoni,
If I go with option 2, will it be a resolution for both of these issues...
1.ERROR TailReader -File will not be read, seekptr checksum did not match
2.INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file
option 2
[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
initCrcLength = 2000
index=testenv
If possible can you explain a bit on this offset numbers..?
INFO WatchedFile - Will begin reading at offset=0 for file='file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log'.
INFO WatchedFile - Will begin reading at offset=90 for file='/opt/app/ws/server/ra_JVM00/log/server.log'.
INFO WatchedFile - Will begin reading at offset=180 for file='file=/opt/app/ws/server/kv_JVM00/log/responseTime.2016-05-04.log'
If your log files presents on NFS file system then you might be hitting this issue for reindexing logs https://answers.splunk.com/answers/130729/splunk-reindexing-files-when-using-remote-shared-filesyste...