Getting Data In

splunk indexes rotated logs

unix
New Member

Example of actual inputs.conf

[monitor:////data/example/server/example/log/*.log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
crcSalt =

first we had an inputs.conf like this:

[monitor:////data/example/server/example/log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
whitelist=(.log$)
crcSalt =

With both configuration splunk indexes rotated logfiles like server.log, server.log.1 etc.
How can I disable indexing rotated logs?

0 Karma
1 Solution

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

mikelanghorst
Motivator

Set a whitelist to only monitor files ending in ,log

0 Karma

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

Ayn
Legend

If you know that, you know the cause of your troubles. What checksum error are you referring to?

0 Karma

unix
New Member

yes I know that, but we need the crcSalt because of the checksum-error...
is there a way to use both?

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...