Shuttl's own S3 integration - QA community help

[Petter_Eriksson]

Hey Shuttl community,

The S3 integration with Shuttl is really the worst part of Shuttl, and maybe also the most wanted feature.
The reason for the S3 integration being so bad, is because Shuttl uses Hadoop's S3 FileSystem implementation, and Hadoop has a very outdated version of their S3 libraries. (https://issues.apache.org/jira/browse/HADOOP-8136, https://issues.apache.org/jira/browse/HADOOP-9454)

If I were to implement my own S3 FileSystem implementation, using updated and faster libraries. Could I ask the community to help me QA my implementation?
I've used Splunk interns to QA in the past, but I don't have those resources anymore.

Having our own S3 integration would also mean that we can implement our own integration features with S3 as soon as they come out. Like security, multi-part uploads and continue canceled upload.

Respond below if you're interested!

Thanks a lot,

- Petter, Shuttl Developer

[Petter_Eriksson]

Hey guys!
Yes, the Shuttl development has stalled a bit since I've been busy with other work.
However, the development branch on Github should support 6.0 and other features as well. I just never came around to test and release it.
It'd be really nice if we could get everything in Develop into the Master branch. That'd make the development of an S3 filesystem smoother as well. It should be releasable. Care to verify that it works with 6.0?

[marksnelling]

I'd also be happy to help. It seems that all development on Shuttl has stalled, it's not even flagged as supporting Splunk 6.0!

[sbaryakov]

Petter,
Was there any progress on that? I would be happy to help.
-Stefan