I am not sure if out of box solution there for log rotation, when you are adding the file or directory. We are using Apache and configured logs with timestamp. I am facing problem in adding these files through add monitor command. I don't know which is effective way I can use to add log files. same time it should not read old files. I heard about white-listing and black listing, But haven't explore much on that. how splunk will read the file when it rotates by its own.
here are the examples of my log file httpaccess.log.2011-01-31, httpaccess.log.2011-05-04
I am using as mention below
./splunk add mointor
Looking forward for reply.
as written in the docs http://www.splunk.com/base/Documentation/latest/Data/MonitorFilesAndDirectories
Splunk detects log file rotation and does not process renamed files it has already indexed (with the exception of .tar and .gz archives; for more information see "Log file rotation" in this manual).
so if you just add the directory where your logs are stored, splunk will take care of it.
I agree with that, but do I need to add files like this way
Also, I do have lot of old files so do I need to put them blacklist?
just add the path where your logs are stored, no need to add every single file. also no need to exclude old files, once indexed by splunk old file are 'ignored'.
I don't have license for that big logs, my question is simple, is the regular expression for adding these files are correct, if so I can customized by expresssion to avoid those files.
BTW, The above mention regular expression is not working for me 😞