How to monitor files with archiving policies

[Zane]

I am currently encountering a problem where I have a log file that will be archived to another folder after reaching certain conditions. I have set up UF monitoring for both files, but the data collected may be duplicate. However, if I do not monitor the archive folder, some logs in the later positions will be lost in the file. I suspect it may be related to the file being archived too quickly? How to solve this problem

for example，my log file is abc.log, and then, it will be archived to current path /debug/abc.1.log, I have set the monitor for both files, but the data is duplicate, however,  if i do not monitor current path /debug/abc.1.log, i will lose the content at the end of the file.

[gcusello]

Hi @Zane,

you could put under monitoring both the folders.

If you don't use the crcSal = <SOURCE> option, Splunk read only the last events in the rotated file and doesn't index twice the logs event if tey have a different filename (Remember that the above option must to be not present!).

Otherwise, if you rename the file before rotating (adding e.g. the new data to the file name), you can delay the rotation (30/60 seconds are sufficient) so Splunk will read also the last event in the file before i's moved to the new folder.

Ciao.

Giuseppe

[Zane]

Hi  @gcusello 

thanks for your answer,

but I can‘t control delayed the rotation, due to those log file not managed by us, so if it's possible，adjusting from the Splunk side would be great.

so as you said, "you can delay the rotation (30/60 seconds are sufficient) so Splunk will read also the last event in the file",  according to this, i found there is a parameter in inputs.conf, "time_before_close", it's 3 by default,  can i adjust this value so as to delay UF close monitored files？for example, set it as 30?

thanks so much.

\Zane

[gcusello]

Hi @Zane,

no, this parameter is when you rotate a file on the same folder, but, if I correctly understood, yu move it in another folder.

The solution is to put under monitoring also the destination folder to tale only the events between the last read and the rotation, remembering that you cannot use crcSalt = <SOUCE> in your inputs.conf.

Ciao.

Giuseppe