Archive
Highlighted

How to move uploaded file to directory splunk is monitoring?

Builder

My splunk instance monitored the directory where proxy server upload compressed access log to via ftp. However my splunk instance sometimes indexed event twice, it result in duplicate events.

I got answer by splunk engineer that Splunk tries hard to read uncompleted file. Sometimes it fails to read. Other time splunk might be able to read. Upload the file to one directory where Splunk is not monitoring, and move it to the directory splunk is monitoring.

My question is if anybody have experience to move the uploaded file to directory splunk is monitoring, please share your experience with me. I think splunk can not do it, I will need to achieve it by using os command or script. I would like to know which os command or what script you used and move the file safely.

Tags (1)
0 Karma
Highlighted

Re: How to move uploaded file to directory splunk is monitoring?

SplunkTrust
SplunkTrust

Check this post for information related to Splunk and atomic operations. http://answers.splunk.com/questions/6482/appending-vs-overwriting-tailed-log-files

This is something you will have to implement outside of Splunk proper, but is manageable as long as you make all of your operations filesystem-atomic

0 Karma