topic Re: Syslog-NG Configuration: Log Rotation Best Practice in Getting Data In

Syslog-NG Configuration: Log Rotation Best Practice

0xAli — Sun, 10 May 2026 07:47:24 GMT

Hi Everyone,

While using Syslog-NG to monitor network traffic and write it into file, I want to ask about the Log file rotation policy:

what is the best practice Hourly Or Daily?
Is that affecting the UF ingestion performance?

# High-volume Palo Alto → hourly
/var/log/firewalls/PaloAlto/$SOURCEIP/$YEAR-$MONTH-$DAY-$HOUR.log

# Lower-volume Fortigate → daily
/var/log/firewalls/Fortigate/$SOURCEIP/$YEAR-$MONTH-$DAY.log

Re: Syslog-NG Configuration: Log Rotation Best Practice

livehybrid — Sun, 10 May 2026 13:59:57 GMT

Hi @0xAli

It really depends on the volume of traffic and the growth rate of the file. I certainly dont think you should expect to see degradation for going with Hourly which I'd probably choose personally - its not going to be millions of files to track and will keep the files smaller to rotate etc.

Would be interested to see what others think though too!

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Syslog-NG Configuration: Log Rotation Best Practice

gcusello — Mon, 11 May 2026 08:14:01 GMT

Hi @0xAli ,

as also @livehybrid pointed on, it depends on the syslog data volume and on the available storage on your receiver.

I usually configure hourly rotation and, when I have available storage, I zip logs after 24 hours (removing zipped files) and I maintain zipped files for three days.

If I don't have sufficient storage for this policy I adapt my polity to the available storage.

In addition, I prefer to use rsyslog instead syslog-ng because it's more modern, but it's a choice!

Ciao.

Giuseppe

Re: Syslog-NG Configuration: Log Rotation Best Practice

0xAli — Mon, 11 May 2026 08:56:50 GMT

Thanks for your support!

If possible, could you please share a sample from your rsyslog config as am used to go with the syslog-ng as it's simple configuration.

another question, is the file size affect the performance of the tail reader, and batch reader?

Re: Syslog-NG Configuration: Log Rotation Best Practice

gcusello — Mon, 11 May 2026 09:06:22 GMT

Hi @0xAli ,

I usually use something like this:

input(type="im<protocol>" port="<port>" ruleset="<input_name>") template(name="<template_name>" type="string" string="/data/syslog/<technology>/%fromhost-ip%/%$YEAR%/%$MONTH%/%$DAY%/<technology>_%$HOUR%.log") ruleset(name="<input_name>"){ action(type="omfile" dynaFile="<template_name>" fileOwner="<splunk_user>" fileGroup="<splunk_group>" dirOwner="<splunk_user>" dirGroup="<splunk_group>") }

in a dedicated .conf file.

ciao.

Giuseppe

Re: Syslog-NG Configuration: Log Rotation Best Practice

splunker_ak — Mon, 11 May 2026 09:22:24 GMT

I have configured log rotation on my heavy forwarders in a way that matches the ingestion rate of each source. This setup helps balance performance with storage management.

High ingesting sources → rotated every 1 hour
Low ingesting sources → rotated at 4 hours, 8 hours, 1 day, or longer depending on the data volume
Daily compression → after 24 hours, files are zipped
Retention policy → zipped files are kept for 1 week
Automatic cleanup → after 1 week, files are deleted to prevent storage fill‑up on the heavy forwarder

Re: Syslog-NG Configuration: Log Rotation Best Practice

PickleRick — Mon, 11 May 2026 22:23:48 GMT

Generally the size of a file should not affect software which just writes at the end of the file or reads there. Of course the bigger a file is, the more time you need to read it whole but that's obvious. If you wanted to cache a whole file in memory (why would you do that?) then the size would matter. Maybe if you used some ancient filesystem the size would matter. With a modern OS the file size doesn't really affect reading/writing performance.

BTW, with modern syslog daemons you can write directly to HEC input and skip local files altogether (which takes the PITA of maintaining the log rotation scripts, monitoring free space and so on).