Shuttl archiving errors

[marksnelling]

I'm having a LOT of trouble getting Shuttl to work with my S3 backend. I can see some buckets in there but my shuttle logs are full of failed bucket messages. Looking at the entries in the shuttl index I can see a lot of these messages.

2013-04-30 14:35:59,091 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Unexpected response code 404, expected 200
2013-04-30 14:35:59,092 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Received error response with XML message
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Unexpected response code 404, expected 200
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Received error response with XML message
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Unexpected response code 404, expected 200
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Received error response with XML message
 2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Received error response with XML message
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Received error response with XML message

[Petter_Eriksson · Accepted solution]

Hey!

That's because Shuttl piggybacks on the S3 implementation of the Hadoop project which uses an old version of jets3t. It is used by a lot of people and it should still work, even though you're getting those warnings in your logs.

There are patches to the Hadoop project which uses the latest version of jets3t and it should be in a release very soon.

If it takes too long for the Hadoop release, I might consider implementing my own native S3 support for Shuttl.

Thanks for letting me know that the Splunkbase version doesn't work. Weird. I'll update it in a bit!

[marksnelling]

I'm using v0.8.3.1 from Github. The application download from Splunkbase doesn't work.