Splunk Enterprise

Smartstore seems to work but shows multipart-upload error messages

jeffland
SplunkTrust
SplunkTrust
In our env, we've had a high value for remote.s3.multipart_upload.part_size to fix a bug present in versions prior to 8.1. When I recently reverted the setting to its default, these logs started popping up from time to time:
 
WARN  S3Client [34628 cachemanagerUploadExecutorWorker-0] - command=multipart-upload command=put transactionId=.... rTxnId=...status=completed success=N uri=... statusCode=500 statusDescription="Internal Server Error" payload="<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>...</RequestId><HostId>...</HostId></Error>"
ERROR S3Client [103868 cachemanagerUploadExecutorWorker-0] - multipart-upload command=end transactionId=... rTxnId=... parts=5 uploadId="..." status=completed success=N payload="<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>...</RequestId><HostId>...</HostId></Error>"
I checked, the uploaded file is the same size as the local one. I was also able to confirm the remote and local files have the same CRC, so it appears the upload worked. The file modtime in the S3 console is three seconds before the failed upload error message is recorded. My guess is there was a temporary issue, and the retry succeeded. I can't find any follow-up logs about the bucket id after the error message. Does anyone else observe this, should I just accept these error messages? Is there something I can do about it?
Labels (2)

thormanrd
Path Finder

I'm seeing the same behavior in v8.2.4.  I resolved many of the 5xx response by increasing the timeouts for the connections and uploads, but I'm still seeing a handful of 50s Internal server: Error responses.  No clue what to do next.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...