Continuously updated log files are not being picked up by Splunk

[rchapman2x]

I have a set of long-running processes that are occasionally restarted. They generate a set of "heartbeat" events where only the timestamp of the event changes, but otherwise the same data is repeated. Occasionally they encounter an interesting event and log a bunch of dynamic data, then go back to the "heartbeat" events. The log files start off very similar and very short, but do eventually grow (not too large; < 1mb each). A new log file is started whenever the process restarts, but otherwise the process will use the same log file until it terminates.

It seems like Splunk is great at reading some of the files, but other files it completely ignores. I checked splunkd.log and found this error message matching one of the missing files:

04-06-2022 10:23:49.155 -0700 ERROR TailReader [19680 tailreader0] - File will not be read, is too small to match seekptr checksum (file=...). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

props.conf:

[custom_json]
DATETIME_CONFIG = 
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6QZ
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TZ = UTC
JSON_TRIM_BRACES_IN_ARRAY_NAMES = true

inputs.conf

[monitor://c:\...\logs]
disabled = false
host = MM-IRV-NB33
sourcetype = custom_json
crcSalt = <SOURCE>

I suspect what happened is that TailReader registered an error on this file which may have been legitimate if the file was too small, but then the error was never cleared and so even though the file grew it would never again be touched by Splunk. Does that sound right?

If so, how do I 1) prevent this error from happening again and 2) clear the error so that my existing files can be read into Splunk?

[rchapman2x · Accepted solution]

I found the issue.

The error message didn't make sense: "Last time we saw this initcrc, filename was different."

If I had "crcSalt = <SOURCE>", then the above message should never occur. So I rechecked inputs.conf and found that, in fact, the crcSalt setting was missing. I'm not sure how that happened. Putting it back and refreshing Splunk solved the issue.