Getting Data In

Splunk Reindexes File that gets a new first line when closed

TiagoMatos
Path Finder

Hello,

My problem is simple to explain: I have an app that generates logs that are written whenever a new action is performed.

The problem is, when the session is over, the first line of that log is changed to include the close time of the session, which makes splunk REINDEX everything on the log.

Any ideas?

Thanks

0 Karma

mattymo
Splunk Employee
Splunk Employee

ah yeah, sucks that it's so tight to the beginning of the file...

Pretty much guarantees you can't use splunk's ability to monitor the file in "realtime"..

Also be weary that if you are monitoring a directory on the HF that has thousands of files ( like var/log/*) it will have to keep track of all the files and can impact performance.

I speak from experience and is why i told you the tale of moving data around, cause once splunk indexes you will want to get rid of those files if you can...

- MattyMo
0 Karma

Richfez
SplunkTrust
SplunkTrust

Could you use a crc length of, like, 23? I was eyeballing it, you'll have to count the right number properly! That'll let it use the FIRST date in it as the check for when it's a new file, ignoring the newly written closing log file time, when that happens.

0 Karma

mattymo
Splunk Employee
Splunk Employee

initCRCLength has to be a minimum of 256

😞

- MattyMo
0 Karma

Richfez
SplunkTrust
SplunkTrust

Does anyone know why there's an arbitrary lower limit?

It makes me suspicious when the smallest possible value for a variable is also the default, and that default/smallest value isn't zero. Suspicious of what? I'm not sure, but suspicious of something. 🙂

Why not let the user decide what they need? There are cases where having it bigger is useful, and we've just stumbled across a situation where being able to make it smaller would be vastly simpler, easier and better than any other solution (excepting "fix the logger").

0 Karma

ddrillic
Ultra Champion

docs

initCrcLength =
* This setting adjusts how much of a file Splunk reads before trying to
identify whether it is a file that has already been seen. You may want to
adjust this if you have many files with common headers (comment headers, long
CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You
may wish to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

0 Karma

TiagoMatos
Path Finder

Yes, it works of I can decrease it as low as I want

0 Karma

jkat54
SplunkTrust
SplunkTrust

I think it has to do with how the fishbucket is architected. I have no doubt it could be modified for this specific case but not recommended due to other reasons.

0 Karma

mattymo
Splunk Employee
Splunk Employee

my thoughts exactly...will see what I can find out!

- MattyMo
0 Karma

Richfez
SplunkTrust
SplunkTrust

Thanks!!!!

0 Karma

mattymo
Splunk Employee
Splunk Employee

My hunch that the initCRCLength minimum is due to hashing constraints has been supported by smarter splunkers than I....will still see if I can fish out the verbose reason...cause im a nerd like that...but even still...28 chars would be crazy...

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

my hunch about the minimum crc length being 256 because of the hashing that is used has been validated by smarter people than I.

Because i always want to know more ill see if i can get the official word, but yeah...likely due to ensuring unique hashes

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...