Getting Data In

how to use crcSalt to index similar folders

mataharry
Communicator

I have a centralized server with all my logs per instance

 /var/log/database/hostA/report.log
 /var/log/database/hostA/report.log.1
 /var/log/database/hostA/report.log.2
 /var/log/database/hostB/report.log
 /var/log/database/hostB/report.log.1
 /var/log/database/hostB/report.log.2
 /var/log/database/hostC/report.log
 /var/log/database/hostC/report.log.1
 /var/log/database/hostC/report.log.2
 /var/log/database/hostD/report.log
 /var/log/database/hostD/report.log.1
 /var/log/database/hostD/report.log.2
etc ...

and a monitoring on each

[monitor://var/log/database/hostA/]
sourcetype=report
host=hostA

Except that the splunk monitor consider that all my reports.log are duplicates.B

10-06-2014 18:06:41.767 -0700 ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=/var/log/database/hostA/report.log). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

If I want to use crcSalt=, it will cause all my rotated versions to be indexed.
Any better options ?

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

You can use a static unique crcSalt for each monitor:

  • then all the files under the same stanza will use the same crcSalt and not be reindexed
  • while files in different stanza will have a different crc and be indexed.

    [monitor://var/log/database/hostA/]
    sourcetype=report
    host=hostA
    crcSalt=A
    [monitor://var/log/database/hostB/]
    sourcetype=report
    host=hostAB
    crcSalt=B

PS : It will not work with a generic input like

[monitor://var/log/database/*/]
 sourcetype=report
 hostsegment=4

View solution in original post

yannK
Splunk Employee
Splunk Employee

You can use a static unique crcSalt for each monitor:

  • then all the files under the same stanza will use the same crcSalt and not be reindexed
  • while files in different stanza will have a different crc and be indexed.

    [monitor://var/log/database/hostA/]
    sourcetype=report
    host=hostA
    crcSalt=A
    [monitor://var/log/database/hostB/]
    sourcetype=report
    host=hostAB
    crcSalt=B

PS : It will not work with a generic input like

[monitor://var/log/database/*/]
 sourcetype=report
 hostsegment=4
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! 🌈 In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...