Monitor stanza in inputs.conf file for rotating log files with access_log.todaysdate as the name.

[saad]

I want to monitor a log file which gets created everyday with new day date in its name.

If I configure inputs.conf as below it will monitor all the older log files of previous days as well which will flood my source list, i want only the latest todays logfile to be monitored. Please suggest.

[monitor:////path to direct/access_log.*]
sourcetype = log4j
ignoreOlderThan = 7d
crcSalt = <string>

[saad]

I guess you dint get my querry. I have a log file which get created daily with todays date. for eg todays logfile name is access_log.2021-09-22. A new file will be created tomm "access_log.2021-09-23". I want to monitor only todays file and not all the previous dates file in that directory.

[codebuilder]

Are these OS logs, or application specific?

If specific to your application and you're unable to change how new logs are named (eliminate the timestamp), then you would have to find a workaround. One option would be to configure logrotate so that the OS rotates and compresses your logs daily, then you could blacklist .gz files.

[PickleRick]

From the configuration I suspect it's some java application. Might be a tomcat or jboss app. They are ugly by design and very troublesome to do log rotation properly.

[codebuilder]

Based on the info in your post, Linux is already rotating your logs as it should. So all you really need to do is remove the wildcard and instead specify the file name with extension (access_log.log e.g.).

Splunk will then monitor that file in real time and pick back up on it after the OS rotates it while ignoring any other files in that directory. access_log.log-20210922, access_log.log.1 or access_log.gz would all be ignored, for example.

[saad]

Thank you for your response.

But unfortunately this would not work as i dont have any file with access_log.log (my log file get created daily with that days date)

I have a log file which gets created daily with todays date. for eg todays logfile name is "access_log.2021-09-22". A new file will be created tomm "access_log.2021-09-23". I want to monitor only todays file and not all the previous dates file in that directory.

[PickleRick]

Firstly, 7d is quite low. You should set a value high enough so that your files should never reach, including maintenances, downtimes and so on. But then again maybe in your case 7d is fine.

Anyway, ignoreOlderThan works by checking file attributes (mtime if I remember correctly) so make sure that your files do have this attribute set to the past. If you are unable to meet this condition (because you're exporting the files over the network and the NFS/CIFS/whatever is cheating on the times or you have a file rotation mechanism that does update the attributes every night) I'm afraid your only hope is to craft a regex to blacklist some of the files.

Oh, and btw, it seems like wrong section of the forum. 😉