Syslog-NG Configuration: Log Rotation Best Practice

[0xAli]

Hi Everyone,

While using Syslog-NG to monitor network traffic and write it into file,  I want to ask about the Log file rotation policy:

what is the best practice Hourly Or Daily?
Is that affecting the UF ingestion performance?

# High-volume Palo Alto → hourly
/var/log/firewalls/PaloAlto/$SOURCEIP/$YEAR-$MONTH-$DAY-$HOUR.log

# Lower-volume Fortigate → daily
/var/log/firewalls/Fortigate/$SOURCEIP/$YEAR-$MONTH-$DAY.log

[gcusello]

Hi @0xAli ,

as also @livehybrid pointed on, it depends on the syslog data volume and on the available storage on your receiver.

I usually configure hourly rotation and, when I have available storage, I zip logs after 24 hours (removing zipped files) and I maintain zipped files for three days.

If I don't have sufficient storage for this policy I adapt my polity to the available storage.

In addition, I prefer to use rsyslog instead syslog-ng because it's more modern, but it's a choice!

Ciao.

Giuseppe

[splunker_ak]

I have configured log rotation on my heavy forwarders in a way that matches the ingestion rate of each source. This setup helps balance performance with storage management.

• High ingesting sources → rotated every 1 hour
• Low ingesting sources → rotated at 4 hours, 8 hours, 1 day, or longer depending on the data volume
• Daily compression → after 24 hours, files are zipped
• Retention policy → zipped files are kept for 1 week
• Automatic cleanup → after 1 week, files are deleted to prevent storage fill‑up on the heavy forwarder

[0xAli]

Thanks for your support!

If possible, could you please share a sample from your rsyslog config as am used to go with the syslog-ng as it's simple configuration.

another question, is the file size affect the performance of the tail reader, and batch reader?

[PickleRick]

Generally the size of a file should not affect software which just writes at the end of the file or reads there. Of course the bigger a file is, the more time you need to read it whole but that's obvious. If you wanted to cache a whole file in memory (why would you do that?) then the size would matter. Maybe if you used some ancient filesystem the size would matter. With a modern OS the file size doesn't really affect reading/writing performance.

BTW, with modern syslog daemons you can write directly to HEC input and skip local files altogether (which takes the PITA of maintaining the log rotation scripts, monitoring free space and so on).

[gcusello]

Hi @0xAli ,

I usually use something like this:

input(type="im<protocol>" port="<port>" ruleset="<input_name>")

template(name="<template_name>" type="string"  string="/data/syslog/<technology>/%fromhost-ip%/%$YEAR%/%$MONTH%/%$DAY%/<technology>_%$HOUR%.log")

ruleset(name="<input_name>"){
        action(type="omfile" dynaFile="<template_name>" fileOwner="<splunk_user>" fileGroup="<splunk_group>" dirOwner="<splunk_user>" dirGroup="<splunk_group>")
}

in a dedicated .conf file.

ciao.

Giuseppe

[livehybrid]

Hi @0xAli 

It really depends on the volume of traffic and the growth rate of the file. I certainly dont think you should expect to see degradation for going with Hourly which I'd probably choose personally - its not going to be millions of files to track and will keep the files smaller to rotate etc. 

Would be interested to see what others think though too!

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing