Getting Data In

load compressed files

dmlee
Communicator

Hi,

as we know , before splunk eat a compressed file, splunk will decompress it first then index it.

but, if we have many compressed files under the same directory (ex: ap_20110301.zip, ap_20110302.zip ...) and their original file name are the same (ex:ap.log), what will happen ?

will splunk decompress all those files then index them ? or decompress and index one by one ?

because their original file name are the same , if splunk decompress all of the files at first , it will overwrite existing files (actually, this is what we observed, but we want to make sure).

thanks.

Tags (1)
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

Splunk never actually decompresses the files within archives to a temporary location on disk. Instead we use a library called "libarchive" that allows us to stream through the contents of archives. These streamed contents are then indexed.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

Splunk never actually decompresses the files within archives to a temporary location on disk. Instead we use a library called "libarchive" that allows us to stream through the contents of archives. These streamed contents are then indexed.

dmlee
Communicator

lessons learned, thanks

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...