topic Unpacking and Uploading certain files from an archive automatically/scripted in Getting Data In

Unpacking and Uploading certain files from an archive automatically/scripted

ValentinM — Mon, 31 May 2021 12:01:03 GMT

What is the best way to get data into Splunk from a zip file (files in different subfolders of the zip) in an automated way?

I need to upload certain txt.files from an archive(debug bundle) into my Splunk deployment. The archive gets downloaded on clients that have access to the splunk deployment, and i need a way to automate this process, instead of unpacking the whole archive and then selecting the files i need one by one and uploading them.

Any help appreciated.

Re: Unpacking and Uploading certain files from an archive automatically/scripted

venkatasri — Tue, 01 Jun 2021 02:52:52 GMT

Hi @ValentinM

Splunk does the unpacking automatically, configure the monitor pointing to your .zip file in inputs.conf on the host where files exist. You should have UF/HF or standalone Splunk on the host.

This link helps to config sub-directories - Monitor files and directories with inputs.conf - Splunk Documentation

---------------------------

An upvote would be appreciated if it helps!

Re: Unpacking and Uploading certain files from an archive automatically/scripted

ValentinM — Tue, 01 Jun 2021 08:57:32 GMT

The thing is the .zip file contains multiple hundred files of which i only need about 10 to analyse them in my Splunk Deployment, also sometimes its necessary to download multiple of these .zip files (each zip is a debug bundle of a client pc), and there are multiple users who might download +upload these to Splunk, so if possible i dont want to install a Forwarder on each host.

The overall process should be somewhat like this, a authorirized Splunk User downloads the .zip files he wants to analyse manually, so lets say now he has 4 .zip files on his local machine. Now i need a way in Splunk to unpack the files that i want (about 10 out of the 300+ in the .zip), add 1-2 custom fields at index time to them, and upload them in an automated way.