Getting Data In

Splunk stop sending data after log rotation

splunkjpm
Loves-to-Learn Lots

We have an app input config monitor containing wildcards with whitelist configured to pick up only .log and .out. There are about 120 log files matching the whitelist regex. All the logfiles are ingesting fine except for 1 specific logfile that seems unable to continue the ingestion after log rotation.

crcSalt and initCrcLength already defined as below - 

initCrcLength = 1048576
crcSalt = <SOURCE>

On splunkd.log, the below event can be found  - 

05-15-2024 00:32:57.332 -0400 INFO WatchedFile [16425 tailreader0] - Logfile truncated while open, original pathname file='/xxx/catalina-.out', will begin reading from start.

Is 120 logs on 1 input too many for Splunk to handle? How can we resolve this issue?

Labels (3)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ugh. As I remember from quite a few years back, tomcat logs are awful to deal with.

How are you rotating them? I suppose you're trying logrotate with copytruncate option (because that was the only way that even remotely resembled a "working" solution for rotating this).

The problem I remember from my previous job was that in this case java wouldn't "rewind" the file position pointer and would continue to append to the old file position even though the file got truncated which would mean that you ended up with a sparse file filled with "virtual zeros" up to the previous logfile's end.

catalina.out is a very ugly thing to deal with. As far as I remember, it didn't rotate on its own and if you wanted to "normally" rotate it you'd have to restart your tomcat completely which is a huge PITA.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

it shouldn't bee to much.

Could you show your inputs.conf inside </> block?

Also which UF version and OS you have?

Have you also check that your UF user have access to this new (?) or truncated file?

What 

splunk list inputstatus
splunk list monitor

commands outputs are? Can you find this individual file from those and what status it has?

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...

Splunk AppDynamics Agents Webinar Series

Mark your calendars! On June 24th at 12PM PST, we’re going live with the second session of our Splunk ...