Trying to watch SAP work logs. With some of the development logs, I continually get Checksum for seekptr didn't match, will re-read entire file. Doing a diff on the log files I get an error about the last line not ending with a newline. But diff does just show the new entries. Splunk is currently reindexing the entire files every time there are new lines appended. I have already tried the CHECK_METHOD=modtime and entire_md5, but that causes the files to be completely reindexed every time they change.
diff: Missing newline at the end of file A
diff: Missing newline at the end of file B
Is there a setting or something to force splunk to only read the new lines?
Ok, so I've attempted to figure out what I can about this, maybe it will help you.
The instructions over here talk about using btprobe. I did the checksum for my file, then looked up the details of that particular file. I got something like this:
key=0xa7ca6631a582fe8a scrc=0x6cb7ac65b797477c sptr=22330829 fcrc=0xf6b026bc9c8841b7 flen=0 mdtm=1379005291 wrtm=1379005291
The key is the same as the calculated CRC for the file. I understand that it's "the first half of the MD5 digest of the first 256 bytes of the file". The number of initial bytes might change if you're using the
initCrcLength parameter in 5.0. Anyway, the sptr is the
tell() file position where Splunk left off. Then you take 256 characters back from there to run through MD5 for the scrc (again, taking the first half of the digest string).
I'll note that removing a newline from the end of my file resulted in a different 256 bytes being returned, and of course a different checksum. It could be that whatever is writing the log is confusing the
tell() library call, or is doing something funky with the newline at the end.
so-splunky  tail -c 256 crc-test.log workers=1, qwork_units=0 09-12-2013 10:01:31.201 -0700 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 09-12-2013 10:01:31.201 -0700 INFO Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0 so-splunky  tail -c 256 crc-test-no-newl.log workers=1, qwork_units=0 09-12-2013 10:01:31.201 -0700 INFO Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0 09-12-2013 10:01:31.201 -0700 INFO Metrics - group=tpool, name=indexertpool, qsize=0, workers=6, qwork_units=0
Sorry, I've been swamped and haven't been able to put together a cohesive response about the output from btprobe. I hope to have something for you before the weekend!
(You might also consider filing a support case, if you hadn't already.)
Many of the Splunk-provided binaries have library dependencies that may not be met by your core system. To get around this, you'll want to use the "splunk cmd <X>" notation to invoke btprobe. This runs the binary in the same runtime environment as splunkd, and ensures that the dependencies are satisfied. Try
/opt/splunk/bin/splunk cmd btprobe <args>
So when I run the btprobe command on our AIX 6.1 systems, I get the following.
/opt/splunk> bin/btprobe -help
Could not load program /opt/splunk/bin/splunkd:
Dependent module /dzl41/lib/libxml2.a(libxml2.so.2) could not be loaded.
Member libxml2.so.2 is not found in archive