Universal Forwarder holding open a deleted file

[pkeller]

We had a recent instance where logfiles that were starting to be loaded up with null characters and growing exponentially larger wound up requiring a Splunk restart in order to release them.

The UF was obviously not forwarding them due to the binary nature of the file, but the logs show that the UF continued trying. An SA removed the file, but the UF would not let it go.

Is this expected behavior? I've not encountered this in the past, where log rotation or removal resulted in the process keeping the file held.

splunkd 13880 splunkuser 38r REG 9,2 843716702208 51601434 /home/blah/logs/blah/sfdc/blah.blah-6-3-blah.app1.20150604.NNN.gmt.log (deleted)

The splunkd logs only say:

TailingProcessor - File will not be read, is too small to match seekptr checksum ...

So, if the file won't be read, any ideas as to why the UF wouldn't release it?

[HiroshiSatoh]

I think that it is the 6.6 series bug of UF that is solved in 6.6.4.
https://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4

SPL-142334
logs are delayed in reading after rotation

[vonsolo29]

the UF is version 6.5.3. we aren't seeing this happen to all clients just a few.

[erik_purins]

I still see this with Splunk Universal Forwarder 7.0.2 (build 03bbabbd5c0f) and a rolling log file, wondering if there is some flag or setting I am not using correctly in inputs?

[HiroshiSatoh]

I am sorry that it is useless.
In my environment, UF was a rotate file and the file was OPENed and it was an event that I could not capture logs.Workaround was a restart.

[vonsolo29]

We just had something similar happen. the UF had a hold on some archived files and the file began to grow exponentially until it filled the disk. I couldn't delete the file and had to stop splunk UF and then delete it. Does anyone know why or where i could look to see why splunk had a hold on the files and make sure this doesn't happen again.