This may be Java's fault. Well, "fault" may be a little harsh, but it is a result of how certain versions of the JVM buffer GC log writes versus Splunk's assumptions of when a file is "done".
When the JVM writes these messages, they don't seem to be line-buffered - so a partial line is possible, with a few-second delay until the completion of said line. Splunk reads the log to EOF and sees the (partial) line and saves it in memory, setting a timer of some sort. The JVM doesn't write the rest of the line until after the timer expires so Splunk's assumption is "that's all of that event" and forwards it on with a "done" flag on it. The "done" flag is like an implicit line breaker, which makes for a broken GC log event...
I don't know if there's a way to tune Splunk to "wait longer", or a way to tell the JVM "please line-buffer these" Either would seem to help the issue.
... View more