Deployment Architecture

Indexing specific files inside a zip folder

lohit
Path Finder

HI all ,

I want o index specific files inside a zip folder. Directory structure is like this.

D:/Reports/abc.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv

Also the tool will generate same zip structure i. D:/reports but the zip file name will be different. Rest all the structure beneath the .zip will be same like below

D:/Reports/abc.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv
D:/Reports/123.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv

I have 2 questions

  1. How to index lmn.csv file only.
  2. How to make sure that indexed lmn.csv file should be overwritten as soon as there is a new lmn.csv file genrated by toll in D:/reports.

PLs help !!

Tags (2)

jbsplunk
Splunk Employee
Splunk Employee

You can't index only a particular file within a zip file. I would actually suggest that you not index the data while it's contained within a zip file because the ArchiveProcessor of splunkd is single threaded and can only read one file at a time. This can be a significant bottleneck in terms of performance on the Splunk instance reading this data. The MUCH better alternative would be to unpack the files with a script and monitor the folder that contains the unpacked files. While it might be easier just to monitor the zip files, it really isn't worth the trouble the bottleneck can cause.

If you've got no alternative but to monitor the zip files, then what you can do is to throw away any data that doesn't match a pattern specified using props and transforms by sending it to nullQueue You can find instructions for that here.

Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...