Indexing specific files inside a zip folder

[lohit]

HI all ,

I want o index specific files inside a zip folder. Directory structure is like this.

D:/Reports/abc.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv

Also the tool will generate same zip structure i. D:/reports but the zip file name will be different. Rest all the structure beneath the .zip will be same like below

D:/Reports/abc.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv
D:/Reports/123.zip/x(empty folder)/y(empty folder)/z(empty folder)/abc.txt,pqr.csv,lmn.csv

I have 2 questions

1. How to index lmn.csv file only.
2. How to make sure that indexed lmn.csv file should be overwritten as soon as there is a new lmn.csv file genrated by toll in D:/reports.

PLs help !!

[jbsplunk]

You can't index only a particular file within a zip file. I would actually suggest that you not index the data while it's contained within a zip file because the ArchiveProcessor of splunkd is single threaded and can only read one file at a time. This can be a significant bottleneck in terms of performance on the Splunk instance reading this data. The MUCH better alternative would be to unpack the files with a script and monitor the folder that contains the unpacked files. While it might be easier just to monitor the zip files, it really isn't worth the trouble the bottleneck can cause.

If you've got no alternative but to monitor the zip files, then what you can do is to throw away any data that doesn't match a pattern specified using props and transforms by sending it to nullQueue You can find instructions for that here.