topic Re: md5, crcSalt, gzip, oh my! in Getting Data In

md5, crcSalt, gzip, oh my!

stensonb — Fri, 27 Apr 2012 18:31:09 GMT

Hello Splunkers -

I'm having trouble figuring out how to make the following work.

I get usage files from a popular CDN delivered to me via FTP. These files come in gzipped...but, Splunk is nice and handles all of that wonderfully.
However, on rare occasions, we may have the usage files redelivered from said CDN. When they are redelivered, the contents of the gzip are identical, but the modified time of the gzip is different (bytes 9-12)...causing Splunk to re-index (which doubles my counts for that period)...which is bad.

So, I'm trying to get around splunk using the first/last 256 bytes to determine uniqueness...I'd like to use something like:

CHECK_METHOD=none
crcSalt =

...which would use filename as the ONLY factor when determining uniqueness, but "CHECK_METHOD=none" isn't an options.

Can anybody suggest an alternative approach?

Re: md5, crcSalt, gzip, oh my!

yannK — Fri, 02 Nov 2012 21:44:42 GMT

update since 5.0 you can increase the length of the crc sample :
see
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Inputsconf

initCrcLength = <integer> * This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has already been seen. You may want to adjust this if you have many files with common headers (comment headers, long CSV headers, etc) and recurring filenames. * CAUTION: Improper use of this setting will cause data to be reindexed. You may wish to consult with Splunk Support before adjusting this value - the default is fine for most installations. * Defaults to 256 (bytes). * Must be in the range 256-1048576.

Re: md5, crcSalt, gzip, oh my!

stensonb — Fri, 02 Nov 2012 21:59:43 GMT

Helpful info, but won't actually do what I'm looking for.

The problem with this proposed solution is that the timestamp of the gzip header is within the first 256 bytes of the file....so, this won't work 😞