topic Log file rotation with date in file name in Splunk Search

Log file rotation with date in file name

1234testtest — Mon, 28 Sep 2020 13:03:01 GMT

Hi,
I have to get all (and ONLY) tomcat std out files in D:/Program Files/Apache Software Foundation/Tomcat 6.0/logs. I have the following name of tomcat std out file tomcat6-stdout.2013-01-03.log - and it changes everyday. How do I monitor these files continuosly

My inputs.conf has the stanza below.
[monitor://D:/Program Files/Apache Software Foundation/Tomcat 6.0/logs/tomcat6-stdout*]
disabled = false
followTail = 0
sourcetype = w_std_log_dynamic

I suspect the problem is with log file rotation as if a new tomcat6-stdout with today's date is getting generated, it is not being indexed (not shown in the sources list in the search app).

Thanks and any pointers would be of great help.

Kindly help

Re: Log file rotation with date in file name

alacercogitatus — Fri, 04 Jan 2013 14:03:41 GMT

You could try using a whitelist for the folder. The * in the stanza relates to path, so your monitor was looking for folders instead of files that match the regex.

[monitor://D:/Program Files/Apache Software Foundation/Tomcat 6.0/logs] disabled = false followTail = 0 sourcetype=w_std_log_dynamic whitelist = tomcat6-stdout.*\.log

Re: Log file rotation with date in file name

kallu — Fri, 04 Jan 2013 14:20:49 GMT

How about adding ".log" to the end. Like this

[monitor://D:/Program Files/Apache Software Foundation/Tomcat 6.0/logs/tomcat6-stdout*.log]

This is then exactly like in examples in Splunk docs (=should work)

http://docs.splunk.com/Documentation/Splunk/latest/Data/Specifyinputpathswithwildcards

Ps. I'm not sure if it makes difference but have you tried turning your "/" to "\" if it is Windows?

Re: Log file rotation with date in file name

alacercogitatus — Fri, 04 Jan 2013 14:24:10 GMT

The primary reason for not doing it that way is that the assumption was that this was a Windows system "D:" and the documentation states that that will not work on Windows.

Warning: In Windows, you cannot currently use a wildcard at the root level. For example, this does not work: [monitor://E:\...\foo\*.log] Splunk logs an error and fails to index the desired files. This is a known issue, described in the Known Issues topic of the Release Notes. Look there (http://docs.splunk.com/Documentation/Splunk/5.0.1/ReleaseNotes/KnownIssues) for details on all known issues.

Re: Log file rotation with date in file name

1234testtest — Mon, 28 Sep 2020 13:03:29 GMT

Thank you. surprisingly after a restart of splunk server - this worked -

[monitor://D:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout*]
disabled = false
followTail = 0
sourcetype = w_std_log_dynamic

Re: Log file rotation with date in file name

lukejadamec — Sat, 27 Aug 2016 16:19:56 GMT

This worked for me also. There are many different sourcetypes in this directory, so for this particular sourcetype with the date in the file name is easier to manage as a source with a wildcard than with a whitelist for a directory - I need all files from the directory, but with different sourcetypes.