Getting Data In

Issues with Splunk trying to open a file during a copy process

Splunk Employee
Splunk Employee

Customer is using a script that extracts the logs from the local SQL instance into a text file. When the extraction is complete, the script copies the file over to a folder being monitored by a Splunk forwarder.

The customer is not always seeing the logs come through to the index and looking in splunkd.log on the forwarder they see the following:

05-24-2012 01:07:41.028 +0100 WARN TailingProcessor - Will retry file='f:\SplunkMonitor\WebProxy\system1_99999523.txt' in 10000ms due to: The process cannot access the file because it is being used by another process.

05-24-2012 01:07:51.031 +0100 ERROR TailingProcessor - Ignoring path due to: failed to open for checksum: 'd:\SplunkMonitor\WebProxy\system1_99999523.txt' (The process cannot access the file because it is being used by another process.)

It appears Splunk is trying to read the file whilst it is still being copied over. The file can’t be red because it is locked during the copy process, retries again in 10000ms, fails again because the copy isn’t complete and then gives up.

Is the 10000ms retry period configurable?

Tags (3)

Splunk Employee
Splunk Employee

The 10000ms limit is currently hard-coded.
There is a enhancement ticket currently logged with Splunk:-
SPL-46229 - Add a configurable backoff interval to file monitoring.

It is hoped resolution to this issue using an improved backoff algorithm will be available shortly in a upcoming maintenance release of Splunk 4.3.x

A possible workaround to this issue would be modify the copy script to copy the file to a non monitored directory and call "splunk add oneshot"


Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...