Getting Data In

Does Splunk index gzip files?

Splunk Employee
Splunk Employee

I'd like to index a directory of 50,000 gzip files. The files range in size from 1 KB to 5 MB. Can Splunk monitor these files without first unpacking the gzips?

1 Solution

Splunk Employee
Splunk Employee

The good news is "YES, Splunk can index gzip files as is!" The bad news is, Splunk will monitor these files one at a time, instead of in parallel. Because it is not possible to predict the uncompressed size of a gzip file, Splunk processes these files in sequence for better control of disk allocation. With respect to performance, this is not ideal for handling 50k files so please consider uncompresing them before having Splunk monitor them to take advantage of Splunk's multi-threading file monitoring capabilities.

View solution in original post

Path Finder
0 Karma

Splunk Employee
Splunk Employee

I think we're actually a bit slower than uncompressing the files first (unsure of details) but it's not far off. Mostly uncompressing that much data with the zlib algorithm just takes a lot of CPU.

Splunk Employee
Splunk Employee

The good news is "YES, Splunk can index gzip files as is!" The bad news is, Splunk will monitor these files one at a time, instead of in parallel. Because it is not possible to predict the uncompressed size of a gzip file, Splunk processes these files in sequence for better control of disk allocation. With respect to performance, this is not ideal for handling 50k files so please consider uncompresing them before having Splunk monitor them to take advantage of Splunk's multi-threading file monitoring capabilities.

View solution in original post

New Member

So, what if you don't want it to read a compressed file? Can you compress using a file extension that will prevent Splunk from attempting to index the file... e.g. rotated and compressed log files?

0 Karma

Legend

You can blacklist compressed files in inputs.conf so that they will be ignored:

[monitor:///var/log]
blacklist=(tgz$|zip$)

will ignore all files in /var/log that end with "tgz" or "zip"

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!