Hey Shuttl community,
The S3 integration with Shuttl is really the worst part of Shuttl, and maybe also the most wanted feature.
The reason for the S3 integration being so bad, is because Shuttl uses Hadoop's S3 FileSystem implementation, and Hadoop has a very outdated version of their S3 libraries. (https://issues.apache.org/jira/browse/HADOOP-8136, https://issues.apache.org/jira/browse/HADOOP-9454)
If I were to implement my own S3 FileSystem implementation, using updated and faster libraries. Could I ask the community to help me QA my implementation?
I've used Splunk interns to QA in the past, but I don't have those resources anymore.
Having our own S3 integration would also mean that we can implement our own integration features with S3 as soon as they come out. Like security, multi-part uploads and continue canceled upload.
Respond below if you're interested!
Thanks a lot,
- Petter, Shuttl Developer
... View more