Smartstore : SmartStore throws S3Client 404 error on receipt.json files

[rbal_splunk]

As part of internal testing, migrating data from the Classic index to SmartStore.
The indexes.conf was configured with S3 configurations pointing to the on-prem S3 remote object store, we do see the data getting migrated from the classic index to SmartStore and we can see objects created on the S3 store.

Meanwhile, various 404 errors related to receipt.json file during this process. Interestingly those files do exist on S3 but SmartStore seems to check for this file before uploading it for the first time and at that time the S3 object store throws an error as that object didn't exist on the remote object store.

Here is an excerpt from the splunkd.log

[BserviR]

I'm seeing the same errors. It's looking to perform a GET on an object that doesn’t exist in S3. Did you find a solution?

[sjohnson_splunk]

its benign.. reads before write is known behavior that's why you might see some 404s for bucket uploaded.
https://jira.splunk.com/browse/SPL-171553

[rbal_splunk]

generally, I'd say ignore HTTP status code on their own. Some points around this:

i) There could be intermittent problems in these codes which get resolved by a retry.
ii)The codes become valuable when investigating other issues and that is why we are reporting
iii)They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem.
So if the frequency of the error for a bucket is in ore them, and for example, if it repeats for bucket's receipt.json..

01-04-2019 15:47:13.049 -0800 ERROR S3Client - command=get transactionId=0x7f19c9452000 rTxnId=0x7f19c9431000 status=completed success=N uri=http://URI/splunk-data/_internal/db/3f/9c/5~2BD2B9DE-8F5E-48AD-B566-3BC1ADC9502F/receipt.json statusCode=404 statusDescription="Not Found" payload=""