topic Re: Indexing thousands of zip files in Splunk in Getting Data In

Indexing thousands of zip files in Splunk

ips_mandar — Thu, 06 Jun 2019 07:28:56 GMT

I have one folder where everyday thousands of zip files were added and I want to monitor this folder via Splunk.
So What is best practice to monitor these zip files in splunk and will it impact performance if I monitor zip files directly or shall I extract these file first and then monitor?
Please guide me..

Re: Indexing thousands of zip files in Splunk

arunsundarm — Tue, 03 Sep 2019 08:07:56 GMT

Splunk will be able to handle zip files, however you can extract them and then ingest the data

Re: Indexing thousands of zip files in Splunk

arunsundarm — Thu, 05 Sep 2019 08:28:38 GMT

Just simple mention the source where the zip files needs to be ingested and splunk should decompress and ingest them

Re: Indexing thousands of zip files in Splunk

esix_splunk — Thu, 05 Sep 2019 08:37:41 GMT

Do note that Splunk's unzip process is single threaded. So with 1000s of zip files, you're looking at quite a time to unzip and ingest. You'd be better to unzip the files and use a sinkhole monitor to ingest these file. Much more efficient and will get them indexed much quicker.

Re: Indexing thousands of zip files in Splunk

hunderliggur — Thu, 05 Sep 2019 23:26:37 GMT

It depends by what you mean by monitor and what your resources are. If you are using a Universal Forwarder on the system to monitor the folder I would just configure the UF to input the files as is (zipped).

You do not say if your Splunk Enterprise deployment is a single node (Search+Index on a box) or distributed with seperate search, index, forward nodes. I would first try just monitoring the zip as is and make sure you are not overloading the node doing the input. That would be the simplest/cleanest approach and simplify your operations.