All Apps and Add-ons

Shuttl archiving errors

marksnelling
Communicator

I'm having a LOT of trouble getting Shuttl to work with my S3 backend. I can see some buckets in there but my shuttle logs are full of failed bucket messages. Looking at the entries in the shuttl index I can see a lot of these messages.

2013-04-30 14:35:59,091 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Unexpected response code 404, expected 200
2013-04-30 14:35:59,092 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1343444042_1342777976_4%2FSPLUNK_BUCKET%2Farchive_meta%2Fbucket.size' - Received error response with XML message
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Unexpected response code 404, expected 200
2013-04-30 14:36:00,425 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_5879788359877019897' - Received error response with XML message
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Unexpected response code 404, expected 200
2013-04-30 14:36:17,838 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/block_-5224164744863339667' - Received error response with XML message
 2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,314 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344129802_1343444051_5%2FSPLUNK_BUCKET%2FSources.data' - Received error response with XML message
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Unexpected response code 404, expected 200
2013-04-30 14:36:18,571 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Farchive_root%2Ftemporary_data%2Fmonitor%2Farchive_root%2Farchive_data%2Ftrading%2Fmonitor%2F_blocksignature%2Fdb_1344956552_1344669951_7%2FSPLUNK_BUCKET_TGZ' - Received error response with XML message
Tags (1)
1 Solution

Petter_Eriksson
Splunk Employee
Splunk Employee

Hey!

That's because Shuttl piggybacks on the S3 implementation of the Hadoop project which uses an old version of jets3t. It is used by a lot of people and it should still work, even though you're getting those warnings in your logs.

There are patches to the Hadoop project which uses the latest version of jets3t and it should be in a release very soon.

If it takes too long for the Hadoop release, I might consider implementing my own native S3 support for Shuttl.

Thanks for letting me know that the Splunkbase version doesn't work. Weird. I'll update it in a bit!

View solution in original post

0 Karma

Petter_Eriksson
Splunk Employee
Splunk Employee

Hey!

That's because Shuttl piggybacks on the S3 implementation of the Hadoop project which uses an old version of jets3t. It is used by a lot of people and it should still work, even though you're getting those warnings in your logs.

There are patches to the Hadoop project which uses the latest version of jets3t and it should be in a release very soon.

If it takes too long for the Hadoop release, I might consider implementing my own native S3 support for Shuttl.

Thanks for letting me know that the Splunkbase version doesn't work. Weird. I'll update it in a bit!

0 Karma

marksnelling
Communicator

I'm using v0.8.3.1 from Github. The application download from Splunkbase doesn't work.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...