Getting Data In

Rotate Logs Faster than 5 minutes

fk319
Builder

I am missing logs. My logs rotate faster than 5 minutes, anywhere greater than 1 min.

It seems that every 5 minutes the current log gets loaded, but when it rotates, the new file does not get loaded. If that file gets rotated before the 5 minute mark, then it gets missed.

Has anyone ran into this? is there a work around?
This is on a Universal Forward V5.0.4

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Monitor both the log file and the rotated versions. Splunk can detect that they are rotated versions and finish them once rotated.

example, if your logs are :
/var/log/mylog.log
/var/log/mylog.log.1
/var/log/mylog.log.2
/var/log/mylog.log.3.gz
/var/log/mylog.log.4.gz

in inputs.conf
[monitor:///var/log/mylog*]
sourcetype=mylogsourcetype

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

Monitor both the log file and the rotated versions. Splunk can detect that they are rotated versions and finish them once rotated.

example, if your logs are :
/var/log/mylog.log
/var/log/mylog.log.1
/var/log/mylog.log.2
/var/log/mylog.log.3.gz
/var/log/mylog.log.4.gz

in inputs.conf
[monitor:///var/log/mylog*]
sourcetype=mylogsourcetype

0 Karma

yannK
Splunk Employee
Splunk Employee

Just a remark about speed, if you are using the Universal or Lighweight forwarder, they have a default limit of 256KBps, you can speed them up.
see http://answers.splunk.com/answers/78615/will-this-limit-this-forwarding-speed-to-the-indexer

fk319
Builder

yannK, thanks for your help. It seems that there is a major network issue.

0 Karma

yannK
Splunk Employee
Splunk Employee

Ok, so your log rotation system is doing unconventional things , and splunk considers that the files are different.
- if it's logrotate, is it using copy-truncate ?
- or is it rewriting the first lines of the log while rotating it ?

or are you using the options crcSalt or crcLength ?

0 Karma

fk319
Builder

Did that, and I had multipule copies of my data in Splunk.

0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...