load compressed files

[dmlee]

Hi,

as we know , before splunk eat a compressed file, splunk will decompress it first then index it.

but, if we have many compressed files under the same directory (ex: ap_20110301.zip, ap_20110302.zip ...) and their original file name are the same (ex:ap.log), what will happen ?

will splunk decompress all those files then index them ? or decompress and index one by one ?

because their original file name are the same , if splunk decompress all of the files at first , it will overwrite existing files (actually, this is what we observed, but we want to make sure).

thanks.

[Stephen_Sorkin · Accepted solution]

Splunk never actually decompresses the files within archives to a temporary location on disk. Instead we use a library called "libarchive" that allows us to stream through the contents of archives. These streamed contents are then indexed.

[dmlee]

lessons learned, thanks