Knowledge Management

Need help understanding HTTPS/HTTP requests SmartStore S3 client will send to different buckets from remote store.

jamie00171
Communicator

Hi, 

I'm looking to gain a better understanding of the number of HTTPS/HTTP requests the S3 client will send to both upload buckets and download buckets from the remote store.

When a hot bucket rolls to warm and the S3 client is required to upload the bucket to the remote store does it make a PUT/POST request to the remote store for each file in the bucket or are multiple files batched into the same HTTPS/HTTP request?  I understand that if the file is more than 128MB (assuming the default for remote.s3.multipart_upload.part_size is kept) in size then the upload is broken into multiple requests, but I'm wondering if multiple files are added to the same request up to 128 MB?

Similarly, when a search process requires the objects within a bucket that exists in the remote store is a single HTTP(S) request made for each object/file within the bucket?

Also, it would be great to know if there are any other reasons why the S3 Client would send requests to the remote store other than uploading/downloading a bucket.

My reason for asking is that the pricing for S3 storage on AWS is based on the number of requests: https://aws.amazon.com/s3/pricing/

Many Thanks, 

Jamie

Labels (1)
Tags (3)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

New Release of Federated Search: Bringing Splunk Analytics to More of Your Data

Organizations today are generating more data than ever and storing it across cloud object stores, data lakes, ...

Inside Event Intelligence: How ITSI Turns Network Alerts into Actionable Incidents

Tech Talk Inside Event Intelligence: How ITSI Turns Network Alerts into Actionable Incidents   Correlating ...

Observability Simplified: Combining User Experience, Application Performance & ...

  Tech Talk Network to App: Observability Unlocked   Today’s digital environments span applications, ...