Websphere log monitoring, missing events

[jensmartin]

I am trying to set up monitoring of Websphere logs, they are accessed from a Windows fileshare using CIFS from our linux box that is running Splunk.

The problem is that not all events are logged, I see the following in the splunkd.log:

11-16-2010 12:02:53.554 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/media/was61p2/SystemOut.log'.

11-16-2010 12:02:53.554 INFO WatchedFile - Using follow-tail, will begin reading at EOF for file='/media/was61p2/SystemOut.log'.

Here is the input conf:

[monitor:///media/was61p2/SystemOut.log]
disabled = false
followTail = 1
host = was61p2
sourcetype = websphere_trlog

I have tried both with crcSalt and without, the problem is the same. Anyone have any idea on how to fix this?

[Simon]

First of all, if followTail is activated, Splunk will always begin at the very end of a file when it starts reading one. So after a restart of the forwarder, you'll loose events for sure.

Are you're logs rotating very fast? If yes, it might be a better idea to monitor the hole directory including rotated SystemOut logs, see here http://splunk-base.splunk.com/answers/58549/high-volume-log-rotation for an explanation.

So e.g.:

[monitor:///media/was61p2/]
whitelist = SystemOut.*\.log$
disabled = false
followTail = 1
host = was61p2
sourcetype = websphere_trlog
crcSalt = <SOURCE>

Also make sure, crcSalt is set to . I always had trouble indexing websphere logs without it.

HTH.
Simon