We have an index cluster which utilizes SmartStore in AWS S3. Things appear to be working but we have observed the following in our logs on the index peers.
08-02-2021 12:34:53.725 -0400 ERROR IndexerIf [22952 FilesystemOpExecutorWorker-0] - failed to update bucket bid=sysandy_test2~2~32FED627-479E-41CF-A401-2F061C2EF7E5 with remote metadata due to err= 08-02-2021 12:45:04.668 -0400 ERROR IndexerIf [24603 FilesystemOpExecutorWorker-0] - failed to update bucket bid=sysandy_test2~3~32FED627-479E-41CF-A401-2F061C2EF7E5 with remote metadata due to err= 08-02-2021 12:55:21.684 -0400 ERROR IndexerIf [25930 FilesystemOpExecutorWorker-0] - failed to update bucket bid=sysandy_test2~1~80593238-39FB-443D-8E13-8FC3E521B22C with remote metadata due to err=
It appears that these errors sometimes occur only on one or two of the three index peers but result in a tsidx file that that is different locally then the S3 copy.
It is unclear as to why this is happening and the err= appears to be blank.
Has anyone ever seen this behavior and any suggestions for resolving this ?