Getting Data In

splunk indexes rotated logs

unix
New Member

Example of actual inputs.conf

[monitor:////data/example/server/example/log/*.log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
crcSalt =

first we had an inputs.conf like this:

[monitor:////data/example/server/example/log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
whitelist=(.log$)
crcSalt =

With both configuration splunk indexes rotated logfiles like server.log, server.log.1 etc.
How can I disable indexing rotated logs?

0 Karma
1 Solution

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

mikelanghorst
Motivator

Set a whitelist to only monitor files ending in ,log

0 Karma

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

Ayn
Legend

If you know that, you know the cause of your troubles. What checksum error are you referring to?

0 Karma

unix
New Member

yes I know that, but we need the crcSalt because of the checksum-error...
is there a way to use both?

0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Shape the Future of Splunk: Join the Product Research Lab!

Join the Splunk Product Research Lab and connect with us in the Slack channel #product-research-lab to get ...

Auto-Injector for Everything Else: Making OpenTelemetry Truly Universal

You might have seen Splunk’s recent announcement about donating the OpenTelemetry Injector to the ...