I'm trying to run a batch process for zipped log files. Splunk can read the total number of files (displayed in the Data Input Manager, but it doesn't index them.
Here's my listing of the inputs.conf:
[batch://c:\logs\2010-10\] move_policy = sinkhole crcSalt = <SOURCE> disabled = false
Log file location on HDD:
Thanks for any help!
batch inputs work much the same way as
monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the
archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in
Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.
Here's whats in the splunkd.log:
11-04-2010 18:46:23.346 INFO ArchiveProcessor - handling file=C:\logs\test\2010-10-01\pwdf4619.zip 11-04-2010 18:46:23.346 INFO ArchiveProcessor - reading path=C:\logs\test\2010-10-01\pwdf4619.zip (seek=0 len=2479775) 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - fishstate=key=0xb6863de5131a6b9f sptr=2479775 scrc=0x11837cc8fbac8c86 fnamecrc=0x619903da57a2fbc6 modtime=0 11-04-2010 18:46:23.346 INFO ArchiveProcessor - Finished processing file 'C:\logs\test\2010-10-01\pwdf4619.zip', removing from stats 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - 'C:\logs\test\2010-10-01\pwdf4619.zip' is located in a sinkhole, deleting archive
OK. I fixed this damn issue regarding zip files.
zip files were currupted. Now splunk eats what I drop in that folder.
Mission part 1 accomplished.