topic Splunk stop sending data after log rotation in Getting Data In

Splunk stop sending data after log rotation

splunkjpm — Fri, 17 May 2024 06:04:50 GMT

We have an app input config monitor containing wildcards with whitelist configured to pick up only .log and .out. There are about 120 log files matching the whitelist regex. All the logfiles are ingesting fine except for 1 specific logfile that seems unable to continue the ingestion after log rotation.

crcSalt and initCrcLength already defined as below -

initCrcLength = 1048576
crcSalt = <SOURCE>

On splunkd.log, the below event can be found -

05-15-2024 00:32:57.332 -0400 INFO WatchedFile [16425 tailreader0] - Logfile truncated while open, original pathname file='/xxx/catalina-.out', will begin reading from start.

Is 120 logs on 1 input too many for Splunk to handle? How can we resolve this issue?

Re: Splunk stop sending data after log rotation

isoutamo — Fri, 17 May 2024 14:31:49 GMT

it shouldn't bee to much.

Could you show your inputs.conf inside </> block?

Also which UF version and OS you have?

Have you also check that your UF user have access to this new (?) or truncated file?

What

splunk list inputstatus splunk list monitor

commands outputs are? Can you find this individual file from those and what status it has?

r. Ismo

Re: Splunk stop sending data after log rotation

PickleRick — Fri, 17 May 2024 21:13:37 GMT

Ugh. As I remember from quite a few years back, tomcat logs are awful to deal with.

How are you rotating them? I suppose you're trying logrotate with copytruncate option (because that was the only way that even remotely resembled a "working" solution for rotating this).

The problem I remember from my previous job was that in this case java wouldn't "rewind" the file position pointer and would continue to append to the old file position even though the file got truncated which would mean that you ended up with a sparse file filled with "virtual zeros" up to the previous logfile's end.

catalina.out is a very ugly thing to deal with. As far as I remember, it didn't rotate on its own and if you wanted to "normally" rotate it you'd have to restart your tomcat completely which is a huge PITA.