Getting Data In

What happens if a large log file being monitored hasn't fully been forwarded at the time of rotation?

pkeller
Contributor

If I'm monitoring a very large logfile

[monitor:///home/me/logs]
whitelist = (myApp)\.log$

/home/me/logs/myApp.log

And at some point, a process rotates the file to:

/home/me/logs/OLD/myApp.log

If the file hasn't fully been forwarded at the time of rotation ... will:

  1. myApp.log be monitored in the new directory (assumed because OLD would be in scope for the monitored path)
  2. myApp.log be monitored in its entirety, or will Splunk still know the offset that was last indexed

Thank you.

If

0 Karma
1 Solution

hortonew
Builder

Splunk keeps track of the offset via the fishbucket. Even if the file is moved, it should only index what it hasn't already indexed. So moving it to a different directory shouldn't be a problem.

View solution in original post

0 Karma

hortonew
Builder

Splunk keeps track of the offset via the fishbucket. Even if the file is moved, it should only index what it hasn't already indexed. So moving it to a different directory shouldn't be a problem.

0 Karma
Get Updates on the Splunk Community!

Data Management Digest – December 2025

Welcome to the December edition of Data Management Digest! As we continue our journey of data innovation, the ...

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...