Archive

Indexing thousands of zip files in Splunk

ips_mandar
Builder

I have one folder where everyday thousands of zip files were added and I want to monitor this folder via Splunk.
So What is best practice to monitor these zip files in splunk and will it impact performance if I monitor zip files directly or shall I extract these file first and then monitor?
Please guide me..

Tags (2)
0 Karma

hunderliggur
Path Finder

It depends by what you mean by monitor and what your resources are. If you are using a Universal Forwarder on the system to monitor the folder I would just configure the UF to input the files as is (zipped).

You do not say if your Splunk Enterprise deployment is a single node (Search+Index on a box) or distributed with seperate search, index, forward nodes. I would first try just monitoring the zip as is and make sure you are not overloading the node doing the input. That would be the simplest/cleanest approach and simplify your operations.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Do note that Splunk's unzip process is single threaded. So with 1000s of zip files, you're looking at quite a time to unzip and ingest. You'd be better to unzip the files and use a sinkhole monitor to ingest these file. Much more efficient and will get them indexed much quicker.

arunsundarm
Engager

Splunk will be able to handle zip files, however you can extract them and then ingest the data

0 Karma

arunsundarm
Engager

Just simple mention the source where the zip files needs to be ingested and splunk should decompress and ingest them

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!