Knowledge Management

Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store.

jamie00171
Communicator

Hi, 

I'm looking to gain a better understanding of the number of HTTPS/HTTP requests the S3 client will send to both upload buckets and download buckets from the remote store.

When a hot bucket rolls to warm and the S3 client is required to upload the bucket to the remote store does it make a PUT/POST request to the remote store for each file in the bucket or are multiple files batched into the same HTTPS/HTTP request?  I understand that if the file is more than 128MB (assuming the default for remote.s3.multipart_upload.part_size is kept) in size then the upload is broken into multiple requests, but I'm wondering if multiple files are added to the same request up to 128 MB?

Similarly, when a search process requires the objects within a bucket that exists in the remote store is a single HTTP(S) request made for each object/file within the bucket?

Also, it would be great to know if there are any other reasons why the S3 Client would send requests to the remote store other than uploading/downloading a bucket.

My reason for asking is that the pricing for S3 storage on AWS is based on the number of requests: https://aws.amazon.com/s3/pricing/

Many Thanks, 

Jamie

Labels (1)
Tags (3)
0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...