Getting Data In

Batch large amount of zipped logs

mattmorning
Explorer

I'm trying to run a batch process for zipped log files. Splunk can read the total number of files (displayed in the Data Input Manager, but it doesn't index them.

Here's my listing of the inputs.conf:

[batch://c:\logs\2010-10\] 
move_policy = sinkhole 
crcSalt = <SOURCE> 
disabled = false

Log file location on HDD:

c:\logs\2010-10\2010-10-01\http_access.log

Thanks for any help!

Tags (2)
0 Karma
1 Solution

Mick
Splunk Employee
Splunk Employee

batch inputs work much the same way as monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in splunkd.log

Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.

View solution in original post

mattmorning
Explorer

Here's whats in the splunkd.log:

11-04-2010 18:46:23.346 INFO ArchiveProcessor - handling file=C:\logs\test\2010-10-01\pwdf4619.zip 11-04-2010 18:46:23.346 INFO ArchiveProcessor - reading path=C:\logs\test\2010-10-01\pwdf4619.zip (seek=0 len=2479775) 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - fishstate=key=0xb6863de5131a6b9f sptr=2479775 scrc=0x11837cc8fbac8c86 fnamecrc=0x619903da57a2fbc6 modtime=0 11-04-2010 18:46:23.346 INFO ArchiveProcessor - Finished processing file 'C:\logs\test\2010-10-01\pwdf4619.zip', removing from stats 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - 'C:\logs\test\2010-10-01\pwdf4619.zip' is located in a sinkhole, deleting archive

0 Karma

mattmorning
Explorer

OK. I fixed this damn issue regarding zip files.

zip files were currupted. Now splunk eats what I drop in that folder.

Mission part 1 accomplished.

0 Karma

Mick
Splunk Employee
Splunk Employee

batch inputs work much the same way as monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in splunkd.log

Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.

mattmorning
Explorer

Thank you Mick.

Splunk eats up my files, but there still must be some problems regarding my files.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...