Getting Data In

Rsyslog Log Rotation

dillardo_2
Path Finder

We need to rotate syslog files once they reach a certain size.
Our directory structure looks like the following:

/opt/splunk/syslog/datasource1
/opt/splunk/syslog/datasource2
/opt/splunk/syslog/datasource3

etc.

Once files in the source folders reach 1GB for example, we need them to be moved to /opt/splunk/old_files

Has anyone successfully set up a rotation script?

0 Karma

jawahir007
Path Finder

dillardo_2
Path Finder

Thanks, I did see this and have tried using logrotate. When we rotate the logs, they need to be moved to an archive folder outside of the syslog directory. When I add a command to move the files after postrotate, they files are moved successfully, but rsyslog stops working and must be restarted.

For example: 

/opt/splunk/syslog/*/*
 {
    daily
    missingok
    compress    
    notifempty    
    sharedscripts
    postrotate
                mv /opt/splunk/syslog/*/*.gz /opt/splunk/archive/;
    endscript
}
0 Karma

dillardo_2
Path Finder

This works for us, but rsyslog stops processing data and has to be restarted.

0 Karma

mydog8it
Builder

We create a new file per device type every hour and throw away files in 72 hours.

0 Karma

dillardo_2
Path Finder

@mydog8it, are you able to share your process and the script you are using? That would be helpful.

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

the hourly rotation is by design is you name the log file in rsyslog with mylog-yearmonthday-hour.log
(and that avoid any race condition as it is done directly by rsyslog)
to purge, just add a simple script in /etc/cron.d/purgemylog.cron
with hourly + run as a user who can delete log then run
find /var/log/mylogdir -type f -name \"mylog*.log\"-mtime +2 -delete
if you want to keep them 2 days
(or -mmin +xxx for more granularity)

make sure you specify the directory and a filename form in your find command to avoid any bad surprise...

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...