Deployment Architecture

Universal Forwarder holding open a deleted file

pkeller
Contributor

We had a recent instance where logfiles that were starting to be loaded up with null characters and growing exponentially larger wound up requiring a Splunk restart in order to release them.

The UF was obviously not forwarding them due to the binary nature of the file, but the logs show that the UF continued trying. An SA removed the file, but the UF would not let it go.

Is this expected behavior? I've not encountered this in the past, where log rotation or removal resulted in the process keeping the file held.

splunkd 13880 splunkuser 38r REG 9,2 843716702208 51601434 /home/blah/logs/blah/sfdc/blah.blah-6-3-blah.app1.20150604.NNN.gmt.log (deleted)

The splunkd logs only say:

TailingProcessor - File will not be read, is too small to match seekptr checksum ...

So, if the file won't be read, any ideas as to why the UF wouldn't release it?

HiroshiSatoh
Champion

I think that it is the 6.6 series bug of UF that is solved in 6.6.4.
https://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4

SPL-142334
logs are delayed in reading after rotation

0 Karma

vonsolo29
Explorer

the UF is version 6.5.3. we aren't seeing this happen to all clients just a few.

0 Karma

erik_purins
Explorer

I still see this with Splunk Universal Forwarder 7.0.2 (build 03bbabbd5c0f) and a rolling log file, wondering if there is some flag or setting I am not using correctly in inputs?

0 Karma

HiroshiSatoh
Champion

I am sorry that it is useless.
In my environment, UF was a rotate file and the file was OPENed and it was an event that I could not capture logs.Workaround was a restart.

0 Karma

vonsolo29
Explorer

We just had something similar happen. the UF had a hold on some archived files and the file began to grow exponentially until it filled the disk. I couldn't delete the file and had to stop splunk UF and then delete it. Does anyone know why or where i could look to see why splunk had a hold on the files and make sure this doesn't happen again.

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...