Getting Data In

splunk indexes rotated logs

unix
New Member

Example of actual inputs.conf

[monitor:////data/example/server/example/log/*.log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
crcSalt =

first we had an inputs.conf like this:

[monitor:////data/example/server/example/log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
whitelist=(.log$)
crcSalt =

With both configuration splunk indexes rotated logfiles like server.log, server.log.1 etc.
How can I disable indexing rotated logs?

0 Karma
1 Solution

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

mikelanghorst
Motivator

Set a whitelist to only monitor files ending in ,log

0 Karma

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

Ayn
Legend

If you know that, you know the cause of your troubles. What checksum error are you referring to?

0 Karma

unix
New Member

yes I know that, but we need the crcSalt because of the checksum-error...
is there a way to use both?

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...