Getting Data In
Highlighted

Monitoring timestamp rotation log files

Path Finder

Hi,
I am not sure if out of box solution there for log rotation, when you are adding the file or directory. We are using Apache and configured logs with timestamp. I am facing problem in adding these files through add monitor command. I don't know which is effective way I can use to add log files. same time it should not read old files. I heard about white-listing and black listing, But haven't explore much on that. how splunk will read the file when it rotates by its own.
here are the examples of my log file httpaccess.log.2011-01-31, httpaccess.log.2011-05-04
I am using as mention below
./splunk add mointor /http_access.log.2011-01-31

Looking forward for reply.

0 Karma
Highlighted

Re: Monitoring timestamp rotation log files

SplunkTrust
SplunkTrust

Hi sumitnagal

as written in the docs http://www.splunk.com/base/Documentation/latest/Data/MonitorFilesAndDirectories

Splunk detects log file rotation and does not process renamed files it has already indexed (with the exception of .tar and .gz archives; for more information see "Log file rotation" in this manual).

so if you just add the directory where your logs are stored, splunk will take care of it.

regards

Highlighted

Re: Monitoring timestamp rotation log files

Path Finder

I agree with that, but do I need to add files like this way
add monitor //http_access.log.\d{4}-\d{2}-\d{2}

Also, I do have lot of old files so do I need to put them blacklist?

0 Karma
Highlighted

Re: Monitoring timestamp rotation log files

SplunkTrust
SplunkTrust

just add the path where your logs are stored, no need to add every single file. also no need to exclude old files, once indexed by splunk old file are 'ignored'.

0 Karma
Highlighted

Re: Monitoring timestamp rotation log files

Path Finder

I don't have license for that big logs, my question is simple, is the regular expression for adding these files are correct, if so I can customized by expresssion to avoid those files.
BTW, The above mention regular expression is not working for me 😞

0 Karma
Highlighted

Re: Monitoring timestamp rotation log files

Motivator

Splunk will handle the log rotation. Simply add the monitor with an appropriate wildcard:

./splunk add monitor <path>/http_access.log.*