Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store.

jamie00171 — Fri, 25 Sep 2020 20:35:39 GMT

Hi,

I'm looking to gain a better understanding of the number of HTTPS/HTTP requests the S3 client will send to both upload buckets and download buckets from the remote store.

When a hot bucket rolls to warm and the S3 client is required to upload the bucket to the remote store does it make a PUT/POST request to the remote store for each file in the bucket or are multiple files batched into the same HTTPS/HTTP request? I understand that if the file is more than 128MB (assuming the default for remote.s3.multipart_upload.part_size is kept) in size then the upload is broken into multiple requests, but I'm wondering if multiple files are added to the same request up to 128 MB?

Similarly, when a search process requires the objects within a bucket that exists in the remote store is a single HTTP(S) request made for each object/file within the bucket?

Also, it would be great to know if there are any other reasons why the S3 Client would send requests to the remote store other than uploading/downloading a bucket.

My reason for asking is that the pricing for S3 storage on AWS is based on the number of requests: https://aws.amazon.com/s3/pricing/

Many Thanks,

Jamie

topic Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store. in Knowledge Management

Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store.