topic Re: If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing? in Getting Data In

If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing?

venkat_d — Tue, 03 Mar 2015 00:33:03 GMT

Suppose I am monitoring a file that has 10 lines in it. These lines are already sent to splunk.
Now, if I overwrite this file to have just 5 lines (in other words, my logwriter cleans the log file every time and write some new lines in it - we are reusing the log file that is being monitored)

Will this cause any problems to splunk indexing?

Re: If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing?

NOUMSSI — Wed, 18 Mar 2015 08:39:29 GMT

Hi venkat;

If when you added your file to splunk, you have choose to upload the file, any time that you'll have updates for that file, you'll need to re-indexing the new file for splunk to take care of the update.

But if you choose to monitor the file (continuously index data from file and directory), you'll not need to re-index your file; you'll just need to overwrite the old file by the new file at the same location.
NB: The new file most have the same name like the old file.

In your case, choose the option that correspond to your needs.

Re: If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing?

stephane_cyrill — Fri, 24 Apr 2015 01:04:29 GMT

here is two scenario for you:

The file is deleted or truncated and new data is rewritten from the start; or
The file is written over the beginning with the same old contents up to the point where it was before, then a couple of new lines are added. In the first case, Splunk will have no problems detecting the new data. In the second case, unless the old data is written faster than Splunk can detect that it has been changed/deleted, it will probably wind up double-indexing the old data. If the old file is rewritten fast enough (or moved/renamed over the old one) then there won't be any problems

Re: If I have a monitored log file with lines that are overwritten rather than appended, will this cause any problems to indexing?

stephane_cyrill — Fri, 24 Apr 2015 01:09:27 GMT

www.webadminblog.com/index.php/splunk-best-practices/

take a look on th 4th paragraph.