Getting Data In

File and Directory Monitor

PaulEscher
Explorer

We have several files on many workstations and the files are appended to every few minutes. Instead of having a light forwarder on each workstation we would like to copy the files to a central location every 5 minutes or so. As the file names will stay the same, we will overwrite them with newer files (larger) with each batch copy.

Will Splunk know that it has already indexed the first half of each file? Will this work as expected without duplicate entries? Is there a better way to do this?

Thanks, Paul

Tags (1)
0 Karma
1 Solution

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

View solution in original post

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

bbingham
Builder

A developer for splunk could attest to this better, but from my understanding splunk pulls the first few lines of the file, ties it to the file name and records a hash for the header, then it indexes the file, and remembers the last line, as long as the header is the same, and the last lines are different, splunk marks the file as changed, and will index the remainder of the file, starting from where it left off. I would assume that as long as your header doesn't change with each new copy of the file, splunk wouldn't have an issue picking up where it left off.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...