Checksum for seekptr didn't match, will re-read entire file Checked with diff

[james_hopton]

Trying to watch SAP work logs. With some of the development logs, I continually get Checksum for seekptr didn't match, will re-read entire file. Doing a diff on the log files I get an error about the last line not ending with a newline. But diff does just show the new entries. Splunk is currently reindexing the entire files every time there are new lines appended. I have already tried the CHECK_METHOD=modtime and entire_md5, but that causes the files to be completely reindexed every time they change.

diff: Missing newline at the end of file A
diff: Missing newline at the end of file B

Is there a setting or something to force splunk to only read the new lines?

[sowings]

Ok, so I've attempted to figure out what I can about this, maybe it will help you.

The instructions over here talk about using btprobe. I did the checksum for my file, then looked up the details of that particular file. I got something like this:

key=0xa7ca6631a582fe8a scrc=0x6cb7ac65b797477c sptr=22330829 fcrc=0xf6b026bc9c8841b7 flen=0 mdtm=1379005291 wrtm=1379005291

The key is the same as the calculated CRC for the file. I understand that it's "the first half of the MD5 digest of the first 256 bytes of the file". The number of initial bytes might change if you're using the initCrcLength parameter in 5.0. Anyway, the sptr is the tell() file position where Splunk left off. Then you take 256 characters back from there to run through MD5 for the scrc (again, taking the first half of the digest string).

I'll note that removing a newline from the end of my file resulted in a different 256 bytes being returned, and of course a different checksum. It could be that whatever is writing the log is confusing the tell() library call, or is doing something funky with the newline at the end.

so-splunky [474] tail -c 256 crc-test.log
workers=1, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0

so-splunky [475] tail -c 256 crc-test-no-newl.log
workers=1, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0

[sowings]

In general, support will typically ask for a 'diag' of your Splunk configuration, and in this case, some samples of the login question.

[james_hopton]

I did create a support case a day or two after the answers post, but I am unable to get to the support portal site now. Would it help to upload any files?

[sowings]

Sorry, I've been swamped and haven't been able to put together a cohesive response about the output from btprobe. I hope to have something for you before the weekend!

(You might also consider filing a support case, if you hadn't already.)

[james_hopton]

Oh ok. I still have the problem of splunk rereading the entire file either though it was just appended to. Are there other things I can try?

[sowings]

Many of the Splunk-provided binaries have library dependencies that may not be met by your core system. To get around this, you'll want to use the "splunk cmd <X>" notation to invoke btprobe. This runs the binary in the same runtime environment as splunkd, and ensures that the dependencies are satisfied. Try /opt/splunk/bin/splunk cmd btprobe <args>

[james_hopton]

So when I run the btprobe command on our AIX 6.1 systems, I get the following.

/opt/splunk> bin/btprobe -help
Could not load program /opt/splunk/bin/splunkd:
Dependent module /dzl41/lib/libxml2.a(libxml2.so.2) could not be loaded.
Member libxml2.so.2 is not found in archive

[sowings]

I've got a lead on some info with the 'btprobe' command that I want to follow up on before I attempt an answer. I'll get back to you soon.