Splunk Enterprise

Smartstore seems to work but shows multipart-upload error messages

jeffland
Champion
In our env, we've had a high value for remote.s3.multipart_upload.part_size to fix a bug present in versions prior to 8.1. When I recently reverted the setting to its default, these logs started popping up from time to time:
 
WARN  S3Client [34628 cachemanagerUploadExecutorWorker-0] - command=multipart-upload command=put transactionId=.... rTxnId=...status=completed success=N uri=... statusCode=500 statusDescription="Internal Server Error" payload="<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>...</RequestId><HostId>...</HostId></Error>"
ERROR S3Client [103868 cachemanagerUploadExecutorWorker-0] - multipart-upload command=end transactionId=... rTxnId=... parts=5 uploadId="..." status=completed success=N payload="<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>...</RequestId><HostId>...</HostId></Error>"
I checked, the uploaded file is the same size as the local one. I was also able to confirm the remote and local files have the same CRC, so it appears the upload worked. The file modtime in the S3 console is three seconds before the failed upload error message is recorded. My guess is there was a temporary issue, and the retry succeeded. I can't find any follow-up logs about the bucket id after the error message. Does anyone else observe this, should I just accept these error messages? Is there something I can do about it?
Labels (2)

thormanrd
Path Finder

I'm seeing the same behavior in v8.2.4.  I resolved many of the 5xx response by increasing the timeouts for the connections and uploads, but I'm still seeing a handful of 50s Internal server: Error responses.  No clue what to do next.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...