Splunk Search

Monitor log file inside zip file

krutika_ag
Explorer

Hi All,

 

There are 50 zip files in a folder in those zip folders there are many other files- log/txt/png, out of which I want to monitor a specific log file.

 

Below is the code i have written but it is failing to monitor that log file, please suggest.

[monitor:///home/splunk/*.zip:./WalkbackDetails.log]
disabled = false
index = ziptest

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk cannot monitor a single file within a zip file.  You must monitor the entire zip file or have a script extract the desired file into a monitored location.

---
If this reply helps you, Karma would be appreciated.

inventsekar
SplunkTrust
SplunkTrust

Hi @krutika_ag ... what @richgalloway said was an excellent answer. 

For Splunk newbies, let me rephrase it(the url link for your ref -  https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/Monitorfilesanddirectories) as follows:

How the forwarder monitors archive files

In order to monitor archived files, forwarders decompress archive files, such as a TAR or ZIP file, prior to processing.

Splunk then processes these files in a "single threaded format" (there are pros and cons, but that is a different topic).

The following types of archive files are supported:

  • TAR
  • GZ
  • BZ2
  • TAR.GZ and TGZ
  • TBZ and TBZ2
  • ZIP
  • Z

If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

so, to avoid duplication, you should monitor the whole archive file. 

Lets say if these files are small, then you can monitor the whole archive and the license usage may not be impacted so much (the search time vs index time... should be considered clearly and well planned for this task). 

One more thing to consider:
are you using UF or HF

     --- or both
     ---- or neither(you may directly upload thru SH GUI) - Splunk Support does not support this deployment model) 

 

hope this helped some new Splunkers, thanks. 

krutika_ag
Explorer

Thank You for your reply, I am using both UF and HF

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @krutika_ag 

As per Splunk docs: If you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data. This can result in event duplication.

thus, to avoid duplication, Splunk monitors whole archive files and does not support single file monitoring. 

 

so, you/we can not monitor a single file inside an archive. 

what i would like to suggest you is that, you can ask the developers/app team who creates that archive file to put it in a separate archive file everytime when there is an update to the archive file. 

i am still not much sure of this suggestion, but this should be possible as per my understanding, thanks. 

 

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...