Knowledge Management

Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store.

jamie00171
Communicator

Hi, 

I'm looking to gain a better understanding of the number of HTTPS/HTTP requests the S3 client will send to both upload buckets and download buckets from the remote store.

When a hot bucket rolls to warm and the S3 client is required to upload the bucket to the remote store does it make a PUT/POST request to the remote store for each file in the bucket or are multiple files batched into the same HTTPS/HTTP request?  I understand that if the file is more than 128MB (assuming the default for remote.s3.multipart_upload.part_size is kept) in size then the upload is broken into multiple requests, but I'm wondering if multiple files are added to the same request up to 128 MB?

Similarly, when a search process requires the objects within a bucket that exists in the remote store is a single HTTP(S) request made for each object/file within the bucket?

Also, it would be great to know if there are any other reasons why the S3 Client would send requests to the remote store other than uploading/downloading a bucket.

My reason for asking is that the pricing for S3 storage on AWS is based on the number of requests: https://aws.amazon.com/s3/pricing/

Many Thanks, 

Jamie

Labels (1)
Tags (3)
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...