Getting Data In

Monitoring file being writen on Windows

ikulcsar
Communicator

Hi,

I have to monitor exported events from a remote Windows system. These files are XML files in text format, one XML record per line. Splunk also running on Windows.

Now the events_last.xml file being written, after the rotate, it gets a new name. Monitoring only this events_last.xml file, can I miss some record because of the logrotate?

What do you suggest how to Index these files? Can I monitor events_last.xml, or need to wait for the logrotate?
Should we limit the file size because of Splunk?

Update:
Based on the answers as I understand: It's safe to monitor the directory including the events_last.xml and the rotated files. CRC check will handle it by default.

What if I only monitoring the events_last.xml? The log source continuously writes the file, and when the time comes, it immediately rotates it. Can Splunk catch the last lines? Or rotate process can be faster and Splunk can miss some lines?
I also checked some docs (eg. http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Monitorfilesanddirectories) but did not found detailed one. For [monitor://*] stanza I didn't find any interval attributes.

Update2
Finally, we monitoring only the rotated files. The delta listening didn't works well because of the XML format. The exporter system puts the new logs not absolutely to the end of file (it puts before the last tag), Splunk re-reads the full file, not just the new logs.

Thanks for the help.

Regards,
István

0 Karma

tiagofbmm
Influencer

You can't control how fast Splunk is checking the file for new events. The thing is, Splunk is set up for this case as you can see in the docs: http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Howlogfilerotationishandled

Also I've never seen such a case where a part of a log has been written and the file was immediately rolled so fast Splunk wouldn't detect the new part of log.

0 Karma

tiagofbmm
Influencer

This parameter can also be tuned and useful if you still fear that behaviour could ever happen

time_before_close = <integer>
* Modification time delta required before the file monitor can close a file on
  EOF.
* Tells the system not to close files that have been updated in past <integer>
  seconds.
* Defaults to 3.
0 Karma

tiagofbmm
Influencer

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that

0 Karma

ikulcsar
Communicator

Hi,
I saw once a partially read line but didn't investigate. We suspended the plan to monitor the open file because of the XML root element (Update2).

So far the concluson we cannot monitor an XML file with delta listening.

What should i do now?

0 Karma

deepashri_123
Motivator

Hey ikulscar,

You can monitor both the rotated or non-rotated file and the crc will check-out the hash events.
Refer this link:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Howlogfilerotationishandled

Let me know if this helps!!

0 Karma

tiagofbmm
Influencer

You can just monitor events_last.xml

Splunk user CRC so although it is still monitoring the same file, he registers in the fishbucket the last line he has read, and a Checksum of the beginning and end of file. So when you events_last.xml is cleaned and is starting to be written again, Splunk will read that again automatically.

You won't lose any event.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...