File and Directory Monitor

[PaulEscher]

We have several files on many workstations and the files are appended to every few minutes. Instead of having a light forwarder on each workstation we would like to copy the files to a central location every 5 minutes or so. As the file names will stay the same, we will overwrite them with newer files (larger) with each batch copy.

Will Splunk know that it has already indexed the first half of each file? Will this work as expected without duplicate entries? Is there a better way to do this?

Thanks, Paul

[Michael_Wilde · Accepted solution]

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

[bbingham]

A developer for splunk could attest to this better, but from my understanding splunk pulls the first few lines of the file, ties it to the file name and records a hash for the header, then it indexes the file, and remembers the last line, as long as the header is the same, and the last lines are different, splunk marks the file as changed, and will index the remainder of the file, starting from where it left off. I would assume that as long as your header doesn't change with each new copy of the file, splunk wouldn't have an issue picking up where it left off.