Getting Data In

splunk indexes rotated logs

New Member

Example of actual inputs.conf

[monitor:////data/example/server/example/log/*.log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
crcSalt =

first we had an inputs.conf like this:

[monitor:////data/example/server/example/log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
whitelist=(.log$)
crcSalt =

With both configuration splunk indexes rotated logfiles like server.log, server.log.1 etc.
How can I disable indexing rotated logs?

0 Karma
1 Solution

Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

Motivator

Set a whitelist to only monitor files ending in ,log

0 Karma

Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

Legend

If you know that, you know the cause of your troubles. What checksum error are you referring to?

0 Karma

New Member

yes I know that, but we need the crcSalt because of the checksum-error...
is there a way to use both?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!