Deployment Architecture

Universal Forwarder holding open a deleted file

pkeller
Contributor

We had a recent instance where logfiles that were starting to be loaded up with null characters and growing exponentially larger wound up requiring a Splunk restart in order to release them.

The UF was obviously not forwarding them due to the binary nature of the file, but the logs show that the UF continued trying. An SA removed the file, but the UF would not let it go.

Is this expected behavior? I've not encountered this in the past, where log rotation or removal resulted in the process keeping the file held.

splunkd 13880 splunkuser 38r REG 9,2 843716702208 51601434 /home/blah/logs/blah/sfdc/blah.blah-6-3-blah.app1.20150604.NNN.gmt.log (deleted)

The splunkd logs only say:

TailingProcessor - File will not be read, is too small to match seekptr checksum ...

So, if the file won't be read, any ideas as to why the UF wouldn't release it?

HiroshiSatoh
Champion

I think that it is the 6.6 series bug of UF that is solved in 6.6.4.
https://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4

SPL-142334
logs are delayed in reading after rotation

0 Karma

vonsolo29
Explorer

the UF is version 6.5.3. we aren't seeing this happen to all clients just a few.

0 Karma

erik_purins
Explorer

I still see this with Splunk Universal Forwarder 7.0.2 (build 03bbabbd5c0f) and a rolling log file, wondering if there is some flag or setting I am not using correctly in inputs?

0 Karma

HiroshiSatoh
Champion

I am sorry that it is useless.
In my environment, UF was a rotate file and the file was OPENed and it was an event that I could not capture logs.Workaround was a restart.

0 Karma

vonsolo29
Explorer

We just had something similar happen. the UF had a hold on some archived files and the file began to grow exponentially until it filled the disk. I couldn't delete the file and had to stop splunk UF and then delete it. Does anyone know why or where i could look to see why splunk had a hold on the files and make sure this doesn't happen again.

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...