Getting Data In

ERROR TailReader - File will not be read, seekptr checksum did not match

impurush
Contributor

I am getting the below error because of two files has same first two lines including timestamps in the different folder.

ERROR TailReader - File will not be read, seekptr checksum did not match (file=filename.2021-01-19.txt). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

The monitoring stanza has filename.*.txt.

So if I increase the initcrc or crcSalt, then all the files under the folders will get re-indexed.

Along with crcSalt, I tried to use ignoreOlderThan but still, the old files are getting re-indexed.
Example: ignoreOlderThan=1d, means still yesterday files are getting re-indexed.

Any better solution to prevent this?

Labels (1)
0 Karma

impurush
Contributor

I have done few changes to solve the issue.

  • IgnoreOlderthan=1d
  • blacklist=filename.<yesterday>.txt (Because all the older files are in the same folder structure)
  • Restarted the forwarder exactly at 12o clock

Even though few lines of data got re-indexed, however, it is around 10 to 20 lines only which were acceptable.

0 Karma

manjunathmeti
Champion

Don't increase initCrcLength, if the files are in different folders you can set crcSalt=<SOURCE> for which the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC. 

impurush
Contributor

Thanks @manjunathmeti  for answering the question. 

However, if we put crcSalt=<SOURCE>, then the older files are getting re-indexed because the files are in same folder.

Example:

/app/folderA/locationA/filename_yyyy-mm-dd.txt
/app/folderB/locationB/filename_yyyy-mm-dd.txt

So if we put crcsalt in any of the file, under the location all the files are getting re-indexed. If we put ignoreOlderthan=1d, still the yesterday file is getting re-indexed.

0 Karma

manjunathmeti
Champion

Forwarder reads the file only if system_current_time - file_modification_time > ignoreOlderthan. Check if yesterdays files are still under this window. 

Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...