Hello,
My problem is simple to explain: I have an app that generates logs that are written whenever a new action is performed.
The problem is, when the session is over, the first line of that log is changed to include the close time of the session, which makes splunk REINDEX everything on the log.
Any ideas?
Thanks
ah yeah, sucks that it's so tight to the beginning of the file...
Pretty much guarantees you can't use splunk's ability to monitor the file in "realtime"..
Also be weary that if you are monitoring a directory on the HF that has thousands of files ( like var/log/*) it will have to keep track of all the files and can impact performance.
I speak from experience and is why i told you the tale of moving data around, cause once splunk indexes you will want to get rid of those files if you can...
Could you use a crc length of, like, 23? I was eyeballing it, you'll have to count the right number properly! That'll let it use the FIRST date in it as the check for when it's a new file, ignoring the newly written closing log file time, when that happens.
initCRCLength has to be a minimum of 256
😞
Does anyone know why there's an arbitrary lower limit?
It makes me suspicious when the smallest possible value for a variable is also the default, and that default/smallest value isn't zero. Suspicious of what? I'm not sure, but suspicious of something. 🙂
Why not let the user decide what they need? There are cases where having it bigger is useful, and we've just stumbled across a situation where being able to make it smaller would be vastly simpler, easier and better than any other solution (excepting "fix the logger").
initCrcLength =
* This setting adjusts how much of a file Splunk reads before trying to
identify whether it is a file that has already been seen. You may want to
adjust this if you have many files with common headers (comment headers, long
CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You
may wish to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.
Yes, it works of I can decrease it as low as I want
I think it has to do with how the fishbucket is architected. I have no doubt it could be modified for this specific case but not recommended due to other reasons.
my thoughts exactly...will see what I can find out!
Thanks!!!!
My hunch that the initCRCLength minimum is due to hashing constraints has been supported by smarter splunkers than I....will still see if I can fish out the verbose reason...cause im a nerd like that...but even still...28 chars would be crazy...
my hunch about the minimum crc length being 256 because of the hashing that is used has been validated by smarter people than I.
Because i always want to know more ill see if i can get the official word, but yeah...likely due to ensuring unique hashes