Getting Data In

File and Directory Monitor

PaulEscher
Explorer

We have several files on many workstations and the files are appended to every few minutes. Instead of having a light forwarder on each workstation we would like to copy the files to a central location every 5 minutes or so. As the file names will stay the same, we will overwrite them with newer files (larger) with each batch copy.

Will Splunk know that it has already indexed the first half of each file? Will this work as expected without duplicate entries? Is there a better way to do this?

Thanks, Paul

Tags (1)
0 Karma
1 Solution

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

View solution in original post

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

bbingham
Builder

A developer for splunk could attest to this better, but from my understanding splunk pulls the first few lines of the file, ties it to the file name and records a hash for the header, then it indexes the file, and remembers the last line, as long as the header is the same, and the last lines are different, splunk marks the file as changed, and will index the remainder of the file, starting from where it left off. I would assume that as long as your header doesn't change with each new copy of the file, splunk wouldn't have an issue picking up where it left off.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...