Does a file monitor input work even if the log being monitored is open for writing by the application that manages it?

[andrewtrobec]

Hello all,

As the title states, I'd like to know whether a file input continues to index a log even though that file is open for writing by the application that manages it. I'm busy evaluating whether to keep UFs on source systems with file inputs active, or whether it might be better to externalize those logs through a secondary process and index those to avoid performance issues.

Best regards,

Andrew

[gaurav_maniar]

Hi,

Splunk File Monitoring does not lock the file for writing while indexing the data.
The purpose of Monitoring is to read the files as soon as it gets new data, but it will not lock the file for writing by the application that modifying it.

If there are many source systems, instead of installing Splunk on all the system rather gather all the logs on central system by file transfer or other methods.
Now monitor all the logs from the Central system via Splunk.

[andrewtrobec]

@gaurav_maniar Thanks for the reply! Follow-up question: can a Splunk UF forward new data added to a log even though that log is open for writing by the application that writes to it? Does the application have to release the write log for the Splunk UF to be able to forward new data?

[gaurav_maniar]

As long as the file permission are correct, Splunk will manage modification of files with insert of new lines, without particular configurations. Splunk automatically reads the modified files and forward the newly added logs.

Windows may prevent reading of open files. In that case you can add monitorNoHandle to your file monitor configuration. This Windows-only input lets you read files on Windows systems as Windows writes to them.
For more details - https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/Monitorfilesanddirectories

If the given information answers your all queries, please accept the answer to close the question.