Logs not forwarding after Log Rotation

[sphiwee]

Hi our logs stop forwarding for a while after they have been archived, and this causes us to miss out on valuable data, how can I make splunk start monitoring that log after the old one has been renamed and archived ?

[richgalloway]

Please share the inputs.conf stanza for the files in question.

[sphiwee]

where can I find that file?

[richgalloway]

It's on the forwarder.  Or you can run btool on the forwarder.

splunk btool inputs list | grep -v "system\/default"

Just copy and paste the stanza for the rotated files. 

[sphiwee]

hi

I don't want the rotated log because its being archived, I want the new log that's been generated.. because somehow after log rotation logs stop being forwarded.

[richgalloway]

That is understood.  The solution is to correct your inputs.conf file, but we have to see the current setting to know what needs correcting.

[sphiwee]

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[blacklist:$SPLUNK_HOME/etc/passwd]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]
index = _telemetry
:

[richgalloway]

Which stanza is causing the problem?  IOW, what is the name of the rotated file?

I suspect the [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*] stanza is the cause and the rotated files have an additional extension after ".log" (like ".log.gz", for instance).  If so, the solution is to add a blacklist attribute to the stanza(s) so files with the rotated extension are ignored.

[sphiwee]

are you able to tell me what do these lines do [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*] 

because watchdog.log and cloud.log are not the logs we are monitoring

[richgalloway]

The first of those stanzas monitors files /opt/log/var/log/watchdog directory with names beginning with "watchdog.log".  The second of the stanzas monitors files /opt/log/var/log/splunk directory with names beginning with "splunk_instrumentation_cloud.log".