File no longer monitored

[Branden]

I have a tomcat log file that I was monitoring (catalina.out). This was working fine until it stopped working on October 02, 2010. For some reason, it is no longer being monitored. The file does rotate each night, but I was under the impression that Splunk could handle this- and it did until October 2nd. I am pretty sure I didn't change anything (the operative phrase being "pretty sure", instead of "certain" 🙂 )

Here is the inputs.conf entry from the LightForwarder:

[monitor:///tomcat_base/xyz/logs/catalina.out]
sourcetype=tomcat-catalina-out

The file gets rotated into catalina.out.Y-M-D.log each night at about 8:00 PM.

Here is the transforms.conf entry that was provided to me on here:

[source_clean-YYYY-MM-DD]
# Remove 'YYYY-MM-DD' or 'YYYY_MM_DD' style date from the filename
#   catalina.2009-09-17.log     --> catalina.log
#   quartz-xyz.log.2009-07-08   --> quartz-xyz.log
#   xyz.log.2009-07-15          --> xyz.log
#   xyz_2010-01-01.log          --> xyz.log
#   info_012345_2010_08_17.txt  --> info_012345.txt
DEST_KEY   = MetaData:Source
SOURCE_KEY = MetaData:Source
REGEX    = source::(.*)([-._ ]\d{4}([-_])\d\d\3\d\d)(|\D.*)$
FORMAT   = source::$1$4

Does anyone have any idea why this would stop working?

Thanks!

[Branden]

The data gets written to catalina.out then, each night, the file gets rotated into catalina.Y-M-D.log. I don't monitor those logs because that data was already captured when it was catalina.out.

There actually is no props.conf entry for [tomcat-catalina-out].

[southeringtonp]

Run splunk list monitor and see if you still see that file.

Is there any possibility that binary data could have made it's way into that file?

Run a search for index=_internal catalina.out and look for any warnings or errors.

[Branden]

I did search across "all time" as well. Nothing out of the ordinary. (Good suggestion, though, as that situation did bite me in the rear not long ago with a different search.)

[Branden]

Thank you for your reply.
I don't think any binary data got in there. Not that I can see...
The file is listed when I run "splunk list monitor".
Ah ha... index=_internal catalina.out does have a bunch of these errors:

"ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=/tomcat_base/xyz/logs/catalina.out). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source..."

Hmm.. .There's no way my catalina.out is too small. Or maybe I'm mis-interpreting the error?

Thanks!

[Lowell]

Also, have you tried searching across all time? (Just in case there is some kind of timestamp recognition issue)

[Lowell]

Can you post your props.conf entry for [tomcat-catalina-out] as well? Also, any reason why you aren't monitoring the rotated files too? Sounds like you have a source normalizing transformer in there already, but the monitor stanza doesn't look at the rotated files.