topic Re: Monitor log file inside zip file in Splunk Search

Monitor log file inside zip file

krutika_ag — Sat, 23 Dec 2023 19:11:25 GMT

Hi All,

There are 50 zip files in a folder in those zip folders there are many other files- log/txt/png, out of which I want to monitor a specific log file.

Below is the code i have written but it is failing to monitor that log file, please suggest.

[monitor:///home/splunk/*.zip:./WalkbackDetails.log]
disabled = false
index = ziptest

Re: Monitor log file inside zip file

richgalloway — Sat, 23 Dec 2023 21:18:11 GMT

Splunk cannot monitor a single file within a zip file. You must monitor the entire zip file or have a script extract the desired file into a monitored location.

Re: Monitor log file inside zip file

inventsekar — Sun, 24 Dec 2023 17:40:25 GMT

Hi @krutika_ag ... what @richgalloway said was an excellent answer.

For Splunk newbies, let me rephrase it(the url link for your ref - https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/Monitorfilesanddirectories) as follows:

How the forwarder monitors archive files

In order to monitor archived files, forwarders decompress archive files, such as a TAR or ZIP file, prior to processing.

Splunk then processes these files in a "single threaded format" (there are pros and cons, but that is a different topic).

The following types of archive files are supported:

TAR
GZ
BZ2
TAR.GZ and TGZ
TBZ and TBZ2
ZIP
Z

If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

so, to avoid duplication, you should monitor the whole archive file.

Lets say if these files are small, then you can monitor the whole archive and the license usage may not be impacted so much (the search time vs index time... should be considered clearly and well planned for this task).

One more thing to consider:
are you using UF or HF

--- or both
---- or neither(you may directly upload thru SH GUI) - Splunk Support does not support this deployment model)

hope this helped some new Splunkers, thanks.

Re: Monitor log file inside zip file

krutika_ag — Wed, 27 Dec 2023 09:19:18 GMT

Thank You for your reply, I am using both UF and HF

Re: Monitor log file inside zip file

inventsekar — Mon, 01 Jan 2024 23:48:35 GMT

Hi @krutika_ag

As per Splunk docs: If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

thus, to avoid duplication, Splunk monitors whole archive files and does not support single file monitoring.

so, you/we can not monitor a single file inside an archive.

what i would like to suggest you is that, you can ask the developers/app team who creates that archive file to put it in a separate archive file everytime when there is an update to the archive file.

i am still not much sure of this suggestion, but this should be possible as per my understanding, thanks.