topic Re: What is the best way to use a universal forwarder to monitor a file that overwrites itself at random periods throughout the day? in Getting Data In

What is the best way to use a universal forwarder to monitor a file that overwrites itself at random periods throughout the day?

rfoley — Tue, 28 Jul 2015 21:58:57 GMT

Hoping that someone has seen this before and might be able to help.

I'm fairly new to SPLUNK and I am attempting to send to the indexer a monitored file. This log file overwrites itself throughout the day at random periods based upon the software that generates it. I need to be able to have the file read and the contents pushed to the indexer. I need to make sure that we do not get duplicate values nor miss data from within the file.

We are utilizing SPLUNK 6.1 and the Windows Universal Forwarder to monitor the file. Any insight on the best practice for this scenario would be greatly appreciated.

Thanks.

Re: What is the best way to use a universal forwarder to monitor a file that overwrites itself at random periods throughout the day?

aljohnson_splun — Tue, 28 Jul 2015 23:05:01 GMT

Relevant Questions / Answers:

http://answers.splunk.com/answers/61006/file-system-monitoring-of-text-files-that-are-overwritten.html

http://answers.splunk.com/answers/6482/appending-vs-overwriting-tailed-log-files.html

http://answers.splunk.com/answers/218638/if-i-have-a-monitored-log-file-with-lines-that-are.html

http://answers.splunk.com/answers/2073/when-a-file-is-already-set-to-index-to-splunk-and-that-file-gets-overwritten-with-updates-meaning-instead-of-file-being-appended-does-splunk-smart-enough-to-not-to-reindex-already-indexed-data-and-only-index-whats-newly-aded.html

Re: What is the best way to use a universal forwarder to monitor a file that overwrites itself at random periods throughout the day?

woodcock — Tue, 28 Jul 2015 23:16:38 GMT

A standard file monitor-based inputs.conf entry should work just fine. It may give you some warm fuzzies to use a batch-based entry to have Splunk delete the file once it is forwarded if it comes in all at once (e.g. FTP) .