Batch large amount of zipped logs

[mattmorning]

I'm trying to run a batch process for zipped log files. Splunk can read the total number of files (displayed in the Data Input Manager, but it doesn't index them.

Here's my listing of the inputs.conf:

[batch://c:\logs\2010-10\] 
move_policy = sinkhole 
crcSalt = <SOURCE> 
disabled = false

Log file location on HDD:

c:\logs\2010-10\2010-10-01\http_access.log

Thanks for any help!

[mattmorning]

Here's whats in the splunkd.log:

11-04-2010 18:46:23.346 INFO ArchiveProcessor - handling file=C:\logs\test\2010-10-01\pwdf4619.zip 11-04-2010 18:46:23.346 INFO ArchiveProcessor - reading path=C:\logs\test\2010-10-01\pwdf4619.zip (seek=0 len=2479775) 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - fishstate=key=0xb6863de5131a6b9f sptr=2479775 scrc=0x11837cc8fbac8c86 fnamecrc=0x619903da57a2fbc6 modtime=0 11-04-2010 18:46:23.346 INFO ArchiveProcessor - Finished processing file 'C:\logs\test\2010-10-01\pwdf4619.zip', removing from stats 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - 'C:\logs\test\2010-10-01\pwdf4619.zip' is located in a sinkhole, deleting archive

[mattmorning]

OK. I fixed this damn issue regarding zip files.

zip files were currupted. Now splunk eats what I drop in that folder.

Mission part 1 accomplished.

[Mick · Accepted solution]

batch inputs work much the same way as monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in splunkd.log

Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.

[mattmorning]

Thank you Mick.

Splunk eats up my files, but there still must be some problems regarding my files.