topic Re: Monitoring file being writen on Windows in Getting Data In

Monitoring file being writen on Windows

ikulcsar — Tue, 29 Sep 2020 18:28:25 GMT

Hi,

I have to monitor exported events from a remote Windows system. These files are XML files in text format, one XML record per line. Splunk also running on Windows.

Now the events_last.xml file being written, after the rotate, it gets a new name. Monitoring only this events_last.xml file, can I miss some record because of the logrotate?

What do you suggest how to Index these files? Can I monitor events_last.xml, or need to wait for the logrotate?
Should we limit the file size because of Splunk?

Update:
Based on the answers as I understand: It's safe to monitor the directory including the events_last.xml and the rotated files. CRC check will handle it by default.

What if I only monitoring the events_last.xml? The log source continuously writes the file, and when the time comes, it immediately rotates it. Can Splunk catch the last lines? Or rotate process can be faster and Splunk can miss some lines?
I also checked some docs (eg. http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Monitorfilesanddirectories) but did not found detailed one. For [monitor://*] stanza I didn't find any interval attributes.

Update2
Finally, we monitoring only the rotated files. The delta listening didn't works well because of the XML format. The exporter system puts the new logs not absolutely to the end of file (it puts before the last tag), Splunk re-reads the full file, not just the new logs.

Thanks for the help.

Regards,
István

Re: Monitoring file being writen on Windows

tiagofbmm — Tue, 13 Mar 2018 17:42:59 GMT

You can just monitor events_last.xml

Splunk user CRC so although it is still monitoring the same file, he registers in the fishbucket the last line he has read, and a Checksum of the beginning and end of file. So when you events_last.xml is cleaned and is starting to be written again, Splunk will read that again automatically.

You won't lose any event.

Re: Monitoring file being writen on Windows

deepashri_123 — Tue, 13 Mar 2018 18:26:17 GMT

Hey ikulscar,

You can monitor both the rotated or non-rotated file and the crc will check-out the hash events.
Refer this link:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Howlogfilerotationishandled

Let me know if this helps!!

Re: Monitoring file being writen on Windows

tiagofbmm — Wed, 14 Mar 2018 08:33:07 GMT

You can't control how fast Splunk is checking the file for new events. The thing is, Splunk is set up for this case as you can see in the docs: http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Howlogfilerotationishandled

Also I've never seen such a case where a part of a log has been written and the file was immediately rolled so fast Splunk wouldn't detect the new part of log.

Re: Monitoring file being writen on Windows

tiagofbmm — Wed, 14 Mar 2018 08:37:31 GMT

This parameter can also be tuned and useful if you still fear that behaviour could ever happen

time_before_close = <integer>
* Modification time delta required before the file monitor can close a file on
  EOF.
* Tells the system not to close files that have been updated in past <integer>
  seconds.
* Defaults to 3.

Re: Monitoring file being writen on Windows

tiagofbmm — Wed, 21 Mar 2018 17:04:46 GMT

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that

Re: Monitoring file being writen on Windows

ikulcsar — Mon, 26 Mar 2018 07:12:11 GMT

Hi,
I saw once a partially read line but didn't investigate. We suspended the plan to monitor the open file because of the XML root element (Update2).

So far the concluson we cannot monitor an XML file with delta listening.

What should i do now?