crcSalt = ? if the logfile is archived end of the day

[nareshinsvu]

Hello experts,

My inputs.conf is having below config. Just wondering what happens end of the day? Will my splunk loose the final 10secs updates in the log? Also my logs get rolled and renamed into a different archive directory when they reach 500mb. Will there be a dataloss in splunk during the roll over?

    interval = 10 
    crcSalt = <SOURCE>

[sandyIscream]

@nareshinsvu Can send a sample of your logFile here. As Splunk reads the first and last 256 characters of the monitored file and keep a track of that in a file called fishbucket.

case 1 : If your logfile has a long header and has more than 256 characters. Then I would suggest you to increase the initCrcLength of the monitor stanza. Also please remove the crcSalt in your case.

case 2 : You can add a blacklist property. Which will exclude the file types as .tar.gz or .gz.

You can go through the documentation for input monitor stanza and see which one fits best for your case - https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Inputsconf

Please let me know if it helps!

[sandyIscream]

sample monitor stanza in your case would be something like this

[monitor:///your filepath to be monitor]
index =
sourcetype =
initCrcLength =
blacklist = .(?:txt|gz)$ //It will exclude the .txt file format and .gz

[nareshinsvu]

Hi @sandyIscream - Sorry if this is a silly question to ask - How do we calculate max CRClength for log files?

My log file contains text and JSON data. And it's confidential, so I cant share a sample of it here. Any reason why crcSalt is not recommended? I think It works perfectly for my usecase - (continuously changing logfile which gets archived to a different directory end of the day. And new file gets created at the same location)

I have removed crcSalt and played around with varying initCrcLength. But my events are getting duplicated when a new line is added at the tail of the source log.

[sandyIscream]

Could you please try to add blacklist and add your archived file type, In addition to that also keep your initCrcLength a multiple of 256.

[sandyIscream]

@nareshinsvu did it help solve your problem ?

[nareshinsvu]

No mate. I am not splunking archive directory. So no need to worru about blacklisting. But initCrcLength is not helping me. looks like crcSalt is the only option working for me currently

[harsmarvania57]

Hi,

Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Howlogfilerotationishandled , you'll get idea how splunk handles rolling log files.

Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed.

[nareshinsvu]

Hi @harsmarvania57 - My logfile keeps getting new lines through out the day. What is the alternate option to crcSalt? The log rolls to an archive directory BUT that directory is not read by splunk. My logfile name doesnt get changed at any instance.

[harsmarvania57]

Why do you need crcSalt ? SplunkForwarder reads first 256 Bytes of the log files to check whether that file has already been read and ingested in Splunk. At end of the day when log rotation happens SplunkForwarder checks the same file (here I am assuming file name will be same after rotation) and if it detects different 256 Bytes compare to earlier one then it will start reading file from start.

As you are not monitoring rotated log files, there might be possibility that during log rotation few events will be missed and will not index in splunk.

[nareshinsvu]

I need it because my logfile is continuously monitored and it keeps changing. It only gets archived at the end of the day and a new file will be created with same name under same folder path.
without crcsalt, all my previous log data is getting re-indexed again.