topic Re: growing file in Getting Data In

growing file

splunk_gs — Wed, 13 Jun 2012 15:40:33 GMT

I have a growing file on a remote location. Can i use the forwarder to monitor this file without having to reupload it everytime it grows? i know initially it will upload the main file but overtime when it grows i just want it to add the changes not the entire file. How do i go about setting this up?

Thanks

Re: growing file

sdaniels — Wed, 13 Jun 2012 16:28:41 GMT

Yes, you can install a forwarder on that remote server and set up file monitoring. Splunk will handle log rotation without re-indexing data it has already taken in and remember where it was for a given file so that it only takes in the new data. An exception would be archive files like .tar and .gz. Check the docs.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

Re: growing file

splunk_gs — Wed, 13 Jun 2012 17:31:36 GMT

are there any specific options i would need to select to do this? like tail forwarding?

Re: growing file

sdaniels — Wed, 13 Jun 2012 18:29:24 GMT

You'll want to set a sourcetype and index. There are other settings that you can look at with details in the inputs.conf spec. Splunk will remember where it is in a file to continuously monitor it and not duplicate data.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf