topic Re: Monitoring timestamp rotation log files in Getting Data In

Monitoring timestamp rotation log files

sumitnagal — Mon, 28 Sep 2020 09:31:40 GMT

Hi,
I am not sure if out of box solution there for log rotation, when you are adding the file or directory. We are using Apache and configured logs with timestamp. I am facing problem in adding these files through add monitor command. I don't know which is effective way I can use to add log files. same time it should not read old files. I heard about white-listing and black listing, But haven't explore much on that. how splunk will read the file when it rotates by its own.
here are the examples of my log file http_access.log.2011-01-31, http_access.log.2011-05-04
I am using as mention below
./splunk add mointor /http_access.log.2011-01-31

Looking forward for reply.

Re: Monitoring timestamp rotation log files

MuS — Wed, 04 May 2011 11:07:12 GMT

Hi sumitnagal

as written in the docs http://www.splunk.com/base/Documentation/latest/Data/MonitorFilesAndDirectories

Splunk detects log file rotation and does not process renamed files it has already indexed (with the exception of .tar and .gz archives; for more information see "Log file rotation" in this manual).

so if you just add the directory where your logs are stored, splunk will take care of it.

regards

Re: Monitoring timestamp rotation log files

sumitnagal — Wed, 04 May 2011 11:16:26 GMT

I agree with that, but do I need to add files like this way
add monitor //http_access.log.\d{4}-\d{2}-\d{2}

Also, I do have lot of old files so do I need to put them blacklist?

Re: Monitoring timestamp rotation log files

MuS — Wed, 04 May 2011 11:29:08 GMT

just add the path where your logs are stored, no need to add every single file. also no need to exclude old files, once indexed by splunk old file are 'ignored'.

Re: Monitoring timestamp rotation log files

sumitnagal — Wed, 04 May 2011 11:38:13 GMT

I don't have license for that big logs, my question is simple, is the regular expression for adding these files are correct, if so I can customized by expresssion to avoid those files.
BTW, The above mention regular expression is not working for me 😞

Re: Monitoring timestamp rotation log files

ftk — Wed, 04 May 2011 13:18:53 GMT

Splunk will handle the log rotation. Simply add the monitor with an appropriate wildcard:

./splunk add monitor <path>/http_access.log.*