Issues with Splunk trying to open a file during a copy process

[dshakespeare_sp]

Customer is using a script that extracts the logs from the local SQL instance into a text file. When the extraction is complete, the script copies the file over to a folder being monitored by a Splunk forwarder.

The customer is not always seeing the logs come through to the index and looking in splunkd.log on the forwarder they see the following:

05-24-2012 01:07:41.028 +0100 WARN TailingProcessor - Will retry file='f:\SplunkMonitor\WebProxy\system1_99999523.txt' in 10000ms due to: The process cannot access the file because it is being used by another process.

05-24-2012 01:07:51.031 +0100 ERROR TailingProcessor - Ignoring path due to: failed to open for checksum: 'd:\SplunkMonitor\WebProxy\system1_99999523.txt' (The process cannot access the file because it is being used by another process.)

It appears Splunk is trying to read the file whilst it is still being copied over. The file can’t be red because it is locked during the copy process, retries again in 10000ms, fails again because the copy isn’t complete and then gives up.

Is the 10000ms retry period configurable?

[dshakespeare_sp]

The 10000ms limit is currently hard-coded.
There is a enhancement ticket currently logged with Splunk:-
SPL-46229 - Add a configurable backoff interval to file monitoring.

It is hoped resolution to this issue using an improved backoff algorithm will be available shortly in a upcoming maintenance release of Splunk 4.3.x

A possible workaround to this issue would be modify the copy script to copy the file to a non monitored directory and call "splunk add oneshot"

See
http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/MonitorfilesanddirectoriesusingtheCLI