topic Logs not forwarding after Log Rotation in Splunk Enterprise

Logs not forwarding after Log Rotation

sphiwee — Wed, 18 Nov 2020 08:42:14 GMT

Hi our logs stop forwarding for a while after they have been archived, and this causes us to miss out on valuable data, how can I make splunk start monitoring that log after the old one has been renamed and archived ?

Re: Logs not forwarding after Log Rotation

richgalloway — Wed, 18 Nov 2020 13:54:26 GMT

Please share the inputs.conf stanza for the files in question.

Re: Logs not forwarding after Log Rotation

sphiwee — Wed, 18 Nov 2020 16:13:17 GMT

where can I find that file?

Re: Logs not forwarding after Log Rotation

richgalloway — Wed, 18 Nov 2020 16:33:17 GMT

It's on the forwarder. Or you can run btool on the forwarder.

splunk btool inputs list | grep -v "system\/default"

Just copy and paste the stanza for the rotated files.

Re: Logs not forwarding after Log Rotation

sphiwee — Thu, 19 Nov 2020 07:24:55 GMT

hi

I don't want the rotated log because its being archived, I want the new log that's been generated.. because somehow after log rotation logs stop being forwarded.

Re: Logs not forwarding after Log Rotation

richgalloway — Thu, 19 Nov 2020 13:26:08 GMT

That is understood. The solution is to correct your inputs.conf file, but we have to see the current setting to know what needs correcting.

Re: Logs not forwarding after Log Rotation

sphiwee — Thu, 19 Nov 2020 21:27:20 GMT

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[blacklist:$SPLUNK_HOME/etc/passwd]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]
index = _telemetry
:

Re: Logs not forwarding after Log Rotation

richgalloway — Fri, 20 Nov 2020 14:43:19 GMT

Which stanza is causing the problem? IOW, what is the name of the rotated file?

I suspect the [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*] stanza is the cause and the rotated files have an additional extension after ".log" (like ".log.gz", for instance). If so, the solution is to add a blacklist attribute to the stanza(s) so files with the rotated extension are ignored.

Re: Logs not forwarding after Log Rotation

sphiwee — Mon, 23 Nov 2020 06:33:59 GMT

are you able to tell me what do these lines do [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]

because watchdog.log and cloud.log are not the logs we are monitoring

Re: Logs not forwarding after Log Rotation

richgalloway — Mon, 23 Nov 2020 14:34:27 GMT

The first of those stanzas monitors files /opt/log/var/log/watchdog directory with names beginning with "watchdog.log". The second of the stanzas monitors files /opt/log/var/log/splunk directory with names beginning with "splunk_instrumentation_cloud.log".