Getting Data In

File and Directory Monitor

PaulEscher
Explorer

We have several files on many workstations and the files are appended to every few minutes. Instead of having a light forwarder on each workstation we would like to copy the files to a central location every 5 minutes or so. As the file names will stay the same, we will overwrite them with newer files (larger) with each batch copy.

Will Splunk know that it has already indexed the first half of each file? Will this work as expected without duplicate entries? Is there a better way to do this?

Thanks, Paul

Tags (1)
0 Karma
1 Solution

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

View solution in original post

Michael_Wilde
Splunk Employee
Splunk Employee

A good discussion around how file monitoring works is in Episode 25 - "The Fishbucket List" of the SplunkTalk Podcast.

bbingham
Builder

A developer for splunk could attest to this better, but from my understanding splunk pulls the first few lines of the file, ties it to the file name and records a hash for the header, then it indexes the file, and remembers the last line, as long as the header is the same, and the last lines are different, splunk marks the file as changed, and will index the remainder of the file, starting from where it left off. I would assume that as long as your header doesn't change with each new copy of the file, splunk wouldn't have an issue picking up where it left off.

Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...