Getting Data In

Splunk stop sending data after log rotation

splunkjpm
Loves-to-Learn Lots

We have an app input config monitor containing wildcards with whitelist configured to pick up only .log and .out. There are about 120 log files matching the whitelist regex. All the logfiles are ingesting fine except for 1 specific logfile that seems unable to continue the ingestion after log rotation.

crcSalt and initCrcLength already defined as below - 

initCrcLength = 1048576
crcSalt = <SOURCE>

On splunkd.log, the below event can be found  - 

05-15-2024 00:32:57.332 -0400 INFO WatchedFile [16425 tailreader0] - Logfile truncated while open, original pathname file='/xxx/catalina-.out', will begin reading from start.

Is 120 logs on 1 input too many for Splunk to handle? How can we resolve this issue?

Labels (3)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Ugh. As I remember from quite a few years back, tomcat logs are awful to deal with.

How are you rotating them? I suppose you're trying logrotate with copytruncate option (because that was the only way that even remotely resembled a "working" solution for rotating this).

The problem I remember from my previous job was that in this case java wouldn't "rewind" the file position pointer and would continue to append to the old file position even though the file got truncated which would mean that you ended up with a sparse file filled with "virtual zeros" up to the previous logfile's end.

catalina.out is a very ugly thing to deal with. As far as I remember, it didn't rotate on its own and if you wanted to "normally" rotate it you'd have to restart your tomcat completely which is a huge PITA.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

it shouldn't bee to much.

Could you show your inputs.conf inside </> block?

Also which UF version and OS you have?

Have you also check that your UF user have access to this new (?) or truncated file?

What 

splunk list inputstatus
splunk list monitor

commands outputs are? Can you find this individual file from those and what status it has?

r. Ismo

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...