Getting Data In

Why are universal forwarders reporting "File will not be read, seekptr checksum did not match"?

prakash007
Builder

We're getting bunch of these exceptions on our Universal Forwarders...any help would be appreciated and I can provide more info if needed...

1) ERROR TailReader -File will not be read, seekptr checksum did not matchit says file will not be read. Does that mean it's ignoring the live log which is logged in the path specified..?

ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/jr_LCMI/log/server.log).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info

2) INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file Does that mean it's re-indexing entire file again..?

INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/app/ws/ATG-Data/home/servers/ku_JVM00/logs/apps.log'

 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/app/ws/server/ra_JVM00/log/server.log'.

As soon as I restart UFs, I see the message below. What do the offset numbers mean..?

INFO  WatchedFile - Will begin reading at offset=0 for file='file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log'.
INFO  WatchedFile - Will begin reading at offset=90 for file='/opt/app/ws/server/ra_JVM00/log/server.log'.
INFO  WatchedFile - Will begin reading at offset=180 for file='file=/opt/app/ws/server/kv_JVM00/log/responseTime.2016-05-04.log'.

Configs on my universal forwarders:

inputs.conf

[monitor:///opt/app/ws/server/*/log/server.log]
sourcetype=log4j
index=testenv

[monitor:///opt/app/ws/server/*/log/responseTime.*.log]
Sourcetype=responseTime
index=testenv

[monitor:///opt/app/ws/ATG-Data/home/servers/*/logs/apps.log]
Sourcetype=apps
index=testenv

outputs.conf

#conpressed and useACK were not set for some of the UFs
#splhfserver is a HF which route the data to Splunk, it's not indexing locally..

[tcpout]
defaultGroup = splhfdataforwarder
compressed = true 
useACK = true

[tcpout:splhfdatafowarder]
Server=splhfserver:9997

ddrillic
Ultra Champion

We ended up doing something like -

      [monitor:///opt/app/ws/server/*/log/server.log]
      sourcetype=log4j
      crcSalt = <source>
      initCrcLength = 2000
      index=testenv

prakash007
Builder

when i add the above settings, i still see this messages on splunkd.logs, how did you resolve this...?

File too small to check seekcrc, probably truncated.  Will re-read entire file=...filepath
File too small to check seekcrc, probably truncated.  Will re-read entire file=...filepath
05-05-2016 16:09:54.601 -0500 INFO  WatchedFile - Logfile truncated while open, original pathname file='/opt/app/ws/server/ra_JVM00/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.602 -0500 INFO  WatchedFile - Logfile truncated while open, original pathname ffile='/opt/app/ws/server/ra_JVM04/log/server.log'., will begin reading from start.
05-05-2016 16:09:54.605 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/app/ws/server/ra_JVM02/log/server.log'.
0 Karma

ddrillic
Ultra Champion

So how big is this file? - /opt/app/ws/server/ra_JVM02/log/server.log

0 Karma

prakash007
Builder

this is what i see from the path /opt/app/ws/server/ra_JVM02/log/server.log

$ cat server.log | wc -l

1205
$ cat server.log | wc -c
236896
Adding a crcSalt helped me to get rid of this messages...

 ERROR TailReader -File will not be read, seekptr checksum did not match (file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Adding a initCrcLength = 2000 OR followTail =1, doesn't help me with this issue...not sure how i can verify if this message is a sign of re-indexing or not..?

File too small to check seekcrc, probably truncated.  Will re-read entire file=...filepath
File too small to check seekcrc, probably truncated.  Will re-read entire file=...filepath
05-05-2016 16:09:54.601 -0500 INFO  WatchedFile - Logfile truncated while open, original pathname file='/opt/app/ws/server/ra_JVM00/log/server.log'., will begin reading from start.
 05-05-2016 16:09:54.602 -0500 INFO  WatchedFile - Logfile truncated while open, original pathname ffile='/opt/app/ws/server/ra_JVM04/log/server.log'., will begin reading from start.
 05-05-2016 16:09:54.605 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/app/ws/server/ra_JVM02/log/server.log'.
0 Karma

prakash007
Builder

Will go with the above said option, is there any ways to check if logs are re-indexed or double indexed...?

I did follow this post, but the numbers do not match...

https://answers.splunk.com/answers/24588/how-can-i-check-that-splunk-indexed-the-entire-contents-of-...

0 Karma

ddrillic
Ultra Champion
0 Karma

prakash007
Builder

ddrilic,

If i use a crcSalt on all there sources, is there any chance of re-indexing the data. The answers seems to be relevant to v4.1.5 of splunk.

0 Karma

ddrillic
Ultra Champion

it's applicable to 6.4 - 6.4 - Inputsconf

-- If i use a crcSalt on all there sources, is there any chance of re-indexing the data.
Depends, I guess, on the value of initCrcLength. If it's, let's say, 2000, instead of the default 256, you probably should be ok.

0 Karma

prakash007
Builder

Actually i'm having hard time in figuring it out....

The actual path of the monitor stanza would include..

[monitor:///opt/app/ws/server/*/log/server.log]

/opt/app/ws/server/ra_JVM00/log/server.log
/opt/app/ws/server/ra_JVM01/log/server.log
/opt/app/ws/server/pr_INS00/log/server.log
/opt/app/ws/server/pr_INS02/log/server.log

Can you suggest me which would the best option to go with...

     [monitor:///opt/app/ws/server/*/log/server.log]
     sourcetype=log4j
     crcSalt = <SOURCE>
     index=testenv

OR

     [monitor:///opt/app/ws/server/*/log/server.log]
     sourcetype=log4j
     initCrcLength = 2000
     index=testenv
0 Karma

somesoni2
Revered Legend

Option 2, avoids chances of duplication.

0 Karma

prakash007
Builder

somesoni,
If I go with option 2, will it be a resolution for both of these issues...

1.ERROR TailReader -File will not be read, seekptr checksum did not match
2.INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file

option 2

 [monitor:///opt/app/ws/server/*/log/server.log]
 sourcetype=log4j
 initCrcLength = 2000
 index=testenv

If possible can you explain a bit on this offset numbers..?

INFO  WatchedFile - Will begin reading at offset=0 for file='file=/opt/app/ws/server/kv_JVM01/log/responseTime.2016-05-04.log'.
 INFO  WatchedFile - Will begin reading at offset=90 for file='/opt/app/ws/server/ra_JVM00/log/server.log'.
 INFO  WatchedFile - Will begin reading at offset=180 for file='file=/opt/app/ws/server/kv_JVM00/log/responseTime.2016-05-04.log'
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If your log files presents on NFS file system then you might be hitting this issue for reindexing logs https://answers.splunk.com/answers/130729/splunk-reindexing-files-when-using-remote-shared-filesyste...

0 Karma