Splunk Enterprise

Logs not forwarding after Log Rotation

sphiwee
Contributor

Hi our logs stop forwarding for a while after they have been archived, and this causes us to miss out on valuable data, how can I make splunk start monitoring that log after the old one has been renamed and archived ?

Labels (2)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please share the inputs.conf stanza for the files in question.

---
If this reply helps you, Karma would be appreciated.
0 Karma

sphiwee
Contributor

where can I find that file?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's on the forwarder.  Or you can run btool on the forwarder.

splunk btool inputs list | grep -v "system\/default"

Just copy and paste the stanza for the rotated files. 

---
If this reply helps you, Karma would be appreciated.
0 Karma

sphiwee
Contributor

hi

I don't want the rotated log because its being archived, I want the new log that's been generated.. because somehow after log rotation logs stop being forwarded.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That is understood.  The solution is to correct your inputs.conf file, but we have to see the current setting to know what needs correcting.

---
If this reply helps you, Karma would be appreciated.

sphiwee
Contributor

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup

[blacklist:$SPLUNK_HOME/etc/auth]

[blacklist:$SPLUNK_HOME/etc/passwd]

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*]
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*]
index = _telemetry
:

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Which stanza is causing the problem?  IOW, what is the name of the rotated file?

I suspect the [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*] stanza is the cause and the rotated files have an additional extension after ".log" (like ".log.gz", for instance).  If so, the solution is to add a blacklist attribute to the stanza(s) so files with the rotated extension are ignored.

---
If this reply helps you, Karma would be appreciated.
0 Karma

sphiwee
Contributor

are you able to tell me what do these lines do [monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*] or [monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*] 

because watchdog.log and cloud.log are not the logs we are monitoring

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The first of those stanzas monitors files /opt/log/var/log/watchdog directory with names beginning with "watchdog.log".  The second of the stanzas monitors files /opt/log/var/log/splunk directory with names beginning with "splunk_instrumentation_cloud.log".  

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...