crcSalt issue

[sushildabare]

We have used below monitor in inputs.conf
[monitor:///usr/sap/IXP/SYS/profile/DEFAULT.PFL]

disabled = false

sourcetype = defprof

index = erp

crcSalt = defprof

Few days back I could see the full content of this DEFAULT.PFL file in indexer search.
Suddenly today when I searched the same its showing only first 4-5 lines of this file.

I checked in somefile.out file, but no errors there.

But when I changed the sourcetype and crcSalt string suddenly I could see the full file content in Indexer.
Please let me know what could be the reason for this and why its working after changing the string. This is very strange, kindly help us to fix this issue.

Thanks|Rajashekhar

[BobM · Accepted solution]

CRCSALT is used to make files look different to splunk. Without it, splunk loads the first and last 256 bytes and uses that to create a hash which it then compares with other files. If you define CRCSALT, its value is added before the hash is calculated so the file looks different.

What you have done is add a fixed string "defprof" as the CRCSALT. This will make the file look different once. The next time a similar file arrives, it will be ignored again.

If you use " CRCSALT = <SOURCE> " including <> and capitalisation, splunk will use the source file and folder name which will change for every file. Therefore each file looks diferent.

From inputs.conf.spec

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty. 

[josephka333]

sweet and short, thanks!

[sushildabare]

Hi,
Thanks for your answer.
Before trying the fixed string, we did try "crcSalt = " which did not work as well.
That time also it was showing only first few lines not the full file content.
Moreover the file in question does not get updated very often(may be once/twice a year this file will be modified with few new lines), still splunk is not indexing it.
Without using crcSalt also splunk was not indexing this file.
Please suggest how to fix this.

Thanks
Rajashekhar

[sushildabare]

Can any one please provide your advice on this issue.

[sushildabare]

Please can any one help me on this issue.