Getting Data In

Indexing big zip files

dillencehsu
Path Finder

I have few zip file (after extend is thound of csv files) in a folder, each zip file size is over 1GB.

I use monitor stanza monitor this folder but Splunk did not index these zip file.

 

Splunk 7.3.3 Standalone

 

[monitor://D:\zipfolder]

index =my_index

sourcetype = my_sourcetype

crcSalt = <SOURCE>

 

 

Any suggests ?

Thanks.

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Unzip the files.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

dillencehsu
Path Finder

Yes. Finally, I use shell script unzip thousands of zip files.

I want to know what happened on Splunk.

ArchiveProcess  can not unzip big zip file or unzip for big zip file is take a long time, and Splunk skip it ?
It have any limit for zip file when indexing ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I don't know why Splunk wouldn't read the zip files.  Perhaps, as you suggest, they're too big.  Is there anything in the logs about it?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

dillencehsu
Path Finder

The same logs with each zip file.

 

12-15-2020 14:17:28.450 +0900 INFO ArchiveProcessor - Handling file=D:\logfile.zip
12-15-2020 14:17:28.450 +0900 INFO ArchiveProcessor - reading path=D:\logfile.zip (seek=0 len=1153047505)
12-15-2020 14:17:59.761 +0900 INFO ArchiveProcessor - Finished processing file 'D:\logfile.zip', removing from stats

0 Karma