Monitor log file inside zip file

[krutika_ag]

Hi All,

There are 50 zip files in a folder in those zip folders there are many other files- log/txt/png, out of which I want to monitor a specific log file.

Below is the code i have written but it is failing to monitor that log file, please suggest.

[monitor:///home/splunk/*.zip:./WalkbackDetails.log]
disabled = false
index = ziptest

[richgalloway]

Splunk cannot monitor a single file within a zip file.  You must monitor the entire zip file or have a script extract the desired file into a monitored location.

[inventsekar]

Hi @krutika_ag ... what @richgalloway said was an excellent answer. 

For Splunk newbies, let me rephrase it(the url link for your ref -  https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/Monitorfilesanddirectories) as follows:

How the forwarder monitors archive files

In order to monitor archived files, forwarders decompress archive files, such as a TAR or ZIP file, prior to processing.

Splunk then processes these files in a "single threaded format" (there are pros and cons, but that is a different topic).

The following types of archive files are supported:

• TAR
• GZ
• BZ2
• TAR.GZ and TGZ
• TBZ and TBZ2
• ZIP
• Z

If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

so, to avoid duplication, you should monitor the whole archive file. 

Lets say if these files are small, then you can monitor the whole archive and the license usage may not be impacted so much (the search time vs index time... should be considered clearly and well planned for this task). 

One more thing to consider:
are you using UF or HF

     --- or both
     ---- or neither(you may directly upload thru SH GUI) - Splunk Support does not support this deployment model) 

hope this helped some new Splunkers, thanks. 

[krutika_ag]

Thank You for your reply, I am using both UF and HF

[inventsekar]

Hi @krutika_ag 

As per Splunk docs: If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

thus, to avoid duplication, Splunk monitors whole archive files and does not support single file monitoring. 

so, you/we can not monitor a single file inside an archive. 

what i would like to suggest you is that, you can ask the developers/app team who creates that archive file to put it in a separate archive file everytime when there is an update to the archive file. 

i am still not much sure of this suggestion, but this should be possible as per my understanding, thanks.