topic Re: Does Splunk index gzip files? in Getting Data In

Does Splunk index gzip files?

hulahoop — Sat, 30 Jan 2010 01:47:01 GMT

I'd like to index a directory of 50,000 gzip files. The files range in size from 1 KB to 5 MB. Can Splunk monitor these files without first unpacking the gzips?

Re: Does Splunk index gzip files?

hulahoop — Sat, 30 Jan 2010 01:52:40 GMT

The good news is "YES, Splunk can index gzip files as is!" The bad news is, Splunk will monitor these files one at a time, instead of in parallel. Because it is not possible to predict the uncompressed size of a gzip file, Splunk processes these files in sequence for better control of disk allocation. With respect to performance, this is not ideal for handling 50k files so please consider uncompresing them before having Splunk monitor them to take advantage of Splunk's multi-threading file monitoring capabilities.

Re: Does Splunk index gzip files?

jrodman — Sat, 17 Apr 2010 15:12:05 GMT

I think we're actually a bit slower than uncompressing the files first (unsure of details) but it's not far off. Mostly uncompressing that much data with the zlib algorithm just takes a lot of CPU.

Re: Does Splunk index gzip files?

paulmarino — Tue, 25 Aug 2015 17:38:37 GMT

So, what if you don't want it to read a compressed file? Can you compress using a file extension that will prevent Splunk from attempting to index the file... e.g. rotated and compressed log files?

Re: Does Splunk index gzip files?

hurricanelabs — Fri, 18 Dec 2015 19:27:50 GMT

Here's more details about how that works:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories#How_Splunk_Enterprise_monitors_archived_files

Re: Does Splunk index gzip files?

lguinn2 — Wed, 10 Aug 2016 21:34:22 GMT

You can blacklist compressed files in inputs.conf so that they will be ignored:

[monitor:///var/log]
blacklist=(tgz$|zip$)

will ignore all files in /var/log that end with "tgz" or "zip"