Getting Data In

How to monitor a log file that is getting trimmed at beginning of file

whitepaw00
Explorer

We are using Splunk 9 and are seeing a situation where a file gets re-ingested entirely each time the vendor product trims the older lines from the top of the file. The customer does not have any control over the how the vendor product does the file trimming. Splunk seems to lose track of its pointer and processes each line again even though they have been read previously. This is happening on a Windows client. Any ideas on how to handle this issue?

Labels (1)
0 Karma

dural_yyz
Communicator

https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Inputsconf

# For Windows systems only.
# Does not use file handles

[MonitorNoHandle://<path>]

* This input intercepts file writes to the specific file.
* <path> must be a fully qualified path name to a specific file. Wildcards
  and directories are not accepted.
* This input type does not function on *nix machines.
* You can specify more than one stanza of this type.

disabled = <boolean>
* Whether or not the input is enabled.
* Default: 0 (enabled)

index = <string>
* Specifies the index where this input sends the data.
* This setting is optional.
* Default: the default index

I have no experience using this myself and only learned about this in the last week.  Since you said the UF was on a Windows client this could work for you.  This would not work with a UF on *nix environments. 

whitepaw00
Explorer

Thanks All,

I came up with the same potential fix using MonitorNoHandle. Unfortunately, although I saw no errors in splunkd.log it did not seem to read lines sent into the file.  I followed the examples to set up the stanza and the MonitorNoHandle.exe was started on the server. I will dig deeper to see what might be going on and post what I find here since there is very little about MonitorNoHandle out there today.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk will check the first 256 (configurable) bytes of a monitored file to see if the entire file has been changed rather than new lines added to the end.  If it sees the beginning of the file is different then it assumes the entire file is new and re-ingests it.

I see no workaround for this.  Splunk has no way to know how many older lines were trimmed and so has to treat the whole file as new.

---
If this reply helps you, Karma would be appreciated.

whitepaw00
Explorer

Thanks,

 

Yeah, I read the same which is why I settled on trying out the MonitorNoHandle option. Based on the reading material, it seemed like this would work. 

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...