Hello Splunkers -
I'm having trouble figuring out how to make the following work.
So, I'm trying to get around splunk using the first/last 256 bytes to determine uniqueness...I'd like to use something like:
...which would use filename as the ONLY factor when determining uniqueness, but "CHECK_METHOD=none" isn't an options.
Can anybody suggest an alternative approach?
update since 5.0 you can increase the length of the crc sample :
initCrcLength = <integer>
* This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has
already been seen. You may want to adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You may wish to consult with Splunk
Support before adjusting this value - the default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.
Helpful info, but won't actually do what I'm looking for.
The problem with this proposed solution is that the timestamp of the gzip header is within the first 256 bytes of the file....so, this won't work 😞