Log rotation affecting reading of logs

[sphiwee]

Hi there we noticed we are not getting some logs coming through @ some hours in the morning after log rotation. so we ran the below query.

index=_internal host=* /opt/workfusion/supervisord/log/workfusion.out.log NOT Metrics earliest=-7d latest=now
| timechart span=5m count as NumInt

here's the result below

11-24-2020 01:30:03.080 +0200 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/workfusion/supervisord/log/workfusion.out.log'.

11-19-2020 01:30:04.536 +0200 INFO WatchedFile - Logfile truncated while open, original pathname file='/opt/workfusion/supervisord/log/workfusion.out.log', will begin reading from start.

How can I fix this since it's affecting our dashboard because there are no results or logs so the dashboard is empty. 

[isoutamo]

How you are rotating those logs? mv + touch, cp + truncate something else?

[sphiwee]

cp to backup location, then rm originals 

[isoutamo]

Maybe you should try logrotate to rotate log files e.g. https://linux.die.net/man/8/logrotate
You could try different options how to signal your software to release filehandlers to old log file and star to use new so that you don’t loss events.