After log rotation, UF does not forward logs.

[yutaka1005]

My environment:

Splunk Ver 7.2.3
UF Ver 7.2.3

UF monitors var/log/messages, and forward it to Splunk.
But after log rotation at 02-01-2019 00:05:00, UF no longer forward it.

In internal log, there is message like below.

02-01-2019 00:05:07.503 +0900 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/var/log/messages).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

But I wonder whether there is a possibility that the rotated file will be the same as the first 256 bytes of the already loaded file (the file one generation ago).

Also another weird thing is that there is a message that begins reading the file as follows just before crc error, and the only first 20 lines from the beginning of the rotated file have been indexed in Splunk.

02-01-2019 00:05:04.500 +0900 INFO  WatchedFile - Logfile truncated while open, original pathname file='/var/log/messages', will begin reading from start.

I can't solve it by myself...
If somebody knows about it, tell me...

[woodcock]

By default, Splunk uses the first 256 bytes (the head) to determine if this is a file that it has seen before and the last 256 bytes (the tail) to see if it has changed since the list time that it has seen it. This is stored in the fishbucket (where a fisherman puts the heads and tails which he removes from the fish he catches). Your files have the same first 256 bytes so you need to increase the value of initCrcLength to 256 bytes bigger than the length of the part of the beginning of the file that is constant.

See here:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

In partcular this:

crcSalt = <string>
* Use this setting to force the input to consume files that have matching CRCs
  (cyclic redundancy checks).
    * By default, the input only performs CRC checks against the first 256
      bytes of a file. This behavior prevents the input from indexing the same
      file twice, even though you might have renamed it, as with rolling log
      files, for example. Because the CRC is based on only the first
      few lines of the file, it is possible for legitimately different files
      to have matching CRCs, particularly if they have identical headers.
* If set, <string> is added to the CRC.
* If set to the literal string "<SOURCE>" (including the angle brackets), the
  full directory path to the source file is added to the CRC. This ensures
  that each file being monitored has a unique CRC. When crcSalt is invoked,
  it is usually set to <SOURCE>.
* Be cautious about using this setting with rolling log files; it could lead
  to the log file being re-indexed after it has rolled.
* In many situations, initCrcLength can be used to achieve the same goals.
* Default: empty string.

initCrcLength = <integer>
* How much of a file, in bytes, that the input reads before trying to
  identify whether it is a file that has already been seen. You might want to
  adjust this if you have many files with common headers (comment headers,
  long CSV headers, etc) and recurring filenames.
* Cannot be less than 256 or more than 1048576.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You
  might want to consult with Splunk Support before adjusting this value - the
  default is fine for most installations.
* Default: 256 (bytes).

[yutaka1005]

Thank you for your polite answer!

As lakshman239 answered first, I'm sorry, but accepted that.

[lakshman239 · Accepted solution]

You need to setup crcSalt = in your inputs.conf to force splunk to recalculate CRC checksum and index the file.

Pls refer to https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Howlogfilerotationishandled

[yutaka1005]

Apparently /var/log/messages, which is rotated once a month, coincidentally seemed to have the same leading 256 bytes as in /var/log/messages that was first inputed in last year, because /var/log/messages doesn't have info of year.

So, I could avoid it by setting longer initCRClength.

I wonder if initCRC won't be updated every time the log is rotated ...

[yutaka1005]

Thank you for answer.

But, I think configuring crcSalt is not good idea, because there is below description in Manual.

Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location.

[lakshman239]

try changing only initCrcLength with a higher number say 1024 [ assuming that will show difference when files are rotated for CRC check]