Archive

Checksum for seekptr didn't match, will re-read entire file Checked with diff

New Member

Trying to watch SAP work logs. With some of the development logs, I continually get Checksum for seekptr didn't match, will re-read entire file. Doing a diff on the log files I get an error about the last line not ending with a newline. But diff does just show the new entries. Splunk is currently reindexing the entire files every time there are new lines appended. I have already tried the CHECK_METHOD=modtime and entire_md5, but that causes the files to be completely reindexed every time they change.

diff: Missing newline at the end of file A
diff: Missing newline at the end of file B

Is there a setting or something to force splunk to only read the new lines?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

Ok, so I've attempted to figure out what I can about this, maybe it will help you.

The instructions over here talk about using btprobe. I did the checksum for my file, then looked up the details of that particular file. I got something like this:

key=0xa7ca6631a582fe8a scrc=0x6cb7ac65b797477c sptr=22330829 fcrc=0xf6b026bc9c8841b7 flen=0 mdtm=1379005291 wrtm=1379005291

The key is the same as the calculated CRC for the file. I understand that it's "the first half of the MD5 digest of the first 256 bytes of the file". The number of initial bytes might change if you're using the initCrcLength parameter in 5.0. Anyway, the sptr is the tell() file position where Splunk left off. Then you take 256 characters back from there to run through MD5 for the scrc (again, taking the first half of the digest string).

I'll note that removing a newline from the end of my file resulted in a different 256 bytes being returned, and of course a different checksum. It could be that whatever is writing the log is confusing the tell() library call, or is doing something funky with the newline at the end.

so-splunky [474] tail -c 256 crc-test.log
workers=1, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0

so-splunky [475] tail -c 256 crc-test-no-newl.log
workers=1, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
09-12-2013 10:01:31.201 -0700 INFO  Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0

Splunk Employee
Splunk Employee

In general, support will typically ask for a 'diag' of your Splunk configuration, and in this case, some samples of the login question.

0 Karma

New Member

I did create a support case a day or two after the answers post, but I am unable to get to the support portal site now. Would it help to upload any files?

0 Karma

Splunk Employee
Splunk Employee

Sorry, I've been swamped and haven't been able to put together a cohesive response about the output from btprobe. I hope to have something for you before the weekend!

(You might also consider filing a support case, if you hadn't already.)

0 Karma

New Member

Oh ok. I still have the problem of splunk rereading the entire file either though it was just appended to. Are there other things I can try?

0 Karma

Splunk Employee
Splunk Employee

Many of the Splunk-provided binaries have library dependencies that may not be met by your core system. To get around this, you'll want to use the "splunk cmd <X>" notation to invoke btprobe. This runs the binary in the same runtime environment as splunkd, and ensures that the dependencies are satisfied. Try /opt/splunk/bin/splunk cmd btprobe <args>

0 Karma

New Member

So when I run the btprobe command on our AIX 6.1 systems, I get the following.

/opt/splunk> bin/btprobe -help
Could not load program /opt/splunk/bin/splunkd:
Dependent module /dzl41/lib/libxml2.a(libxml2.so.2) could not be loaded.
Member libxml2.so.2 is not found in archive

0 Karma

Splunk Employee
Splunk Employee

I've got a lead on some info with the 'btprobe' command that I want to follow up on before I attempt an answer. I'll get back to you soon.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!