I'm having trouble figuring out how to make the following work.
I get usage files from a popular CDN delivered to me via FTP. These files come in gzipped...but, Splunk is nice and handles all of that wonderfully.
However, on rare occasions, we may have the usage files redelivered from said CDN. When they are redelivered, the contents of the gzip are identical, but the modified time of the gzip is different (bytes 9-12)...causing Splunk to re-index (which doubles my counts for that period)...which is bad.
So, I'm trying to get around splunk using the first/last 256 bytes to determine uniqueness...I'd like to use something like:
...which would use filename as the ONLY factor when determining uniqueness, but "CHECK_METHOD=none" isn't an options.
initCrcLength = <integer>
* This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has
already been seen. You may want to adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You may wish to consult with Splunk
Support before adjusting this value - the default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.