All Apps and Add-ons

Shuttl archiving errors

marksnelling
Communicator

I'm having a LOT of trouble getting Shuttl to work with my S3 backend. I can see some buckets in there but my shuttle logs are full of failed bucket messages. Looking at the entries in the shuttl index I can see a lot of these messages.

2013-04-30 14:35:59,091 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Unexpected response code 404, expected 200
2013-04-30 14:35:59,092 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Received error response with XML message
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Unexpected response code 404, expected 200
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Received error response with XML message
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Unexpected response code 404, expected 200
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Received error response with XML message
 2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Received error response with XML message
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Received error response with XML message
Tags (1)
1 Solution

Petter_Eriksson
Splunk Employee
Splunk Employee

Hey!

That's because Shuttl piggybacks on the S3 implementation of the Hadoop project which uses an old version of jets3t. It is used by a lot of people and it should still work, even though you're getting those warnings in your logs.

There are patches to the Hadoop project which uses the latest version of jets3t and it should be in a release very soon.

If it takes too long for the Hadoop release, I might consider implementing my own native S3 support for Shuttl.

Thanks for letting me know that the Splunkbase version doesn't work. Weird. I'll update it in a bit!

View solution in original post

0 Karma

Petter_Eriksson
Splunk Employee
Splunk Employee

Hey!

That's because Shuttl piggybacks on the S3 implementation of the Hadoop project which uses an old version of jets3t. It is used by a lot of people and it should still work, even though you're getting those warnings in your logs.

There are patches to the Hadoop project which uses the latest version of jets3t and it should be in a release very soon.

If it takes too long for the Hadoop release, I might consider implementing my own native S3 support for Shuttl.

Thanks for letting me know that the Splunkbase version doesn't work. Weird. I'll update it in a bit!

View solution in original post

0 Karma

marksnelling
Communicator

I'm using v0.8.3.1 from Github. The application download from Splunkbase doesn't work.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!