Getting Data In

crcSalt = ? if the logfile is archived end of the day

nareshinsvu
Builder

Hello experts,

My inputs.conf is having below config. Just wondering what happens end of the day? Will my splunk loose the final 10secs updates in the log? Also my logs get rolled and renamed into a different archive directory when they reach 500mb. Will there be a dataloss in splunk during the roll over?

    interval = 10 
    crcSalt = <SOURCE>
0 Karma

sandyIscream
Communicator

@nareshinsvu Can send a sample of your logFile here. As Splunk reads the first and last 256 characters of the monitored file and keep a track of that in a file called fishbucket.

case 1 : If your logfile has a long header and has more than 256 characters. Then I would suggest you to increase the initCrcLength of the monitor stanza. Also please remove the crcSalt in your case.

case 2 : You can add a blacklist property. Which will exclude the file types as .tar.gz or .gz.

You can go through the documentation for input monitor stanza and see which one fits best for your case - https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Inputsconf

Please let me know if it helps!

0 Karma

sandyIscream
Communicator

sample monitor stanza in your case would be something like this

[monitor:///your filepath to be monitor]
index =
sourcetype =
initCrcLength =
blacklist = .(?:txt|gz)$ //It will exclude the .txt file format and .gz

0 Karma

nareshinsvu
Builder

Hi @sandyIscream - Sorry if this is a silly question to ask - How do we calculate max CRClength for log files?

My log file contains text and JSON data. And it's confidential, so I cant share a sample of it here. Any reason why crcSalt is not recommended? I think It works perfectly for my usecase - (continuously changing logfile which gets archived to a different directory end of the day. And new file gets created at the same location)

I have removed crcSalt and played around with varying initCrcLength. But my events are getting duplicated when a new line is added at the tail of the source log.

0 Karma

sandyIscream
Communicator

Could you please try to add blacklist and add your archived file type, In addition to that also keep your initCrcLength a multiple of 256.

0 Karma

sandyIscream
Communicator

@nareshinsvu did it help solve your problem ?

0 Karma

nareshinsvu
Builder

No mate. I am not splunking archive directory. So no need to worru about blacklisting. But initCrcLength is not helping me. looks like crcSalt is the only option working for me currently

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Howlogfilerotationishandled , you'll get idea how splunk handles rolling log files.

Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed.

0 Karma

nareshinsvu
Builder

Hi @harsmarvania57 - My logfile keeps getting new lines through out the day. What is the alternate option to crcSalt? The log rolls to an archive directory BUT that directory is not read by splunk. My logfile name doesnt get changed at any instance.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Why do you need crcSalt ? SplunkForwarder reads first 256 Bytes of the log files to check whether that file has already been read and ingested in Splunk. At end of the day when log rotation happens SplunkForwarder checks the same file (here I am assuming file name will be same after rotation) and if it detects different 256 Bytes compare to earlier one then it will start reading file from start.

As you are not monitoring rotated log files, there might be possibility that during log rotation few events will be missed and will not index in splunk.

0 Karma

nareshinsvu
Builder

I need it because my logfile is continuously monitored and it keeps changing. It only gets archived at the end of the day and a new file will be created with same name under same folder path.
without crcsalt, all my previous log data is getting re-indexed again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...