How to monitor a log file that is getting trimmed at beginning of file

[whitepaw00]

We are using Splunk 9 and are seeing a situation where a file gets re-ingested entirely each time the vendor product trims the older lines from the top of the file. The customer does not have any control over the how the vendor product does the file trimming. Splunk seems to lose track of its pointer and processes each line again even though they have been read previously. This is happening on a Windows client. Any ideas on how to handle this issue?

[dural_yyz]

https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Inputsconf

# For Windows systems only.
# Does not use file handles

[MonitorNoHandle://<path>]

* This input intercepts file writes to the specific file.
* <path> must be a fully qualified path name to a specific file. Wildcards
  and directories are not accepted.
* This input type does not function on *nix machines.
* You can specify more than one stanza of this type.

disabled = <boolean>
* Whether or not the input is enabled.
* Default: 0 (enabled)

index = <string>
* Specifies the index where this input sends the data.
* This setting is optional.
* Default: the default index

I have no experience using this myself and only learned about this in the last week.  Since you said the UF was on a Windows client this could work for you.  This would not work with a UF on *nix environments. 

[whitepaw00]

Thanks All,

I came up with the same potential fix using MonitorNoHandle. Unfortunately, although I saw no errors in splunkd.log it did not seem to read lines sent into the file.  I followed the examples to set up the stanza and the MonitorNoHandle.exe was started on the server. I will dig deeper to see what might be going on and post what I find here since there is very little about MonitorNoHandle out there today.

[richgalloway]

Splunk will check the first 256 (configurable) bytes of a monitored file to see if the entire file has been changed rather than new lines added to the end.  If it sees the beginning of the file is different then it assumes the entire file is new and re-ingests it.

I see no workaround for this.  Splunk has no way to know how many older lines were trimmed and so has to treat the whole file as new.

[whitepaw00]

Thanks,

Yeah, I read the same which is why I settled on trying out the MonitorNoHandle option. Based on the reading material, it seemed like this would work.