Why am I getting partial results on some indexers?

[kyaparla]

Seeing ERROR message "may have returned partial results" from few indexers".  Logs from those indexers are showing following error messages.

WARN CacheManagerHandler - Localization failure has been reported, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", sid="remote_*_1646235912.747477_6AFBB424-8451-40A4-A05C-A0337BDBC296", errorMessage='waitFor probe, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", did not localize all files before reaching download_status=idle files={"file_types":["tsidx","bloomfilter","deletes"]} local_files={"file_types":["dma_metadata","strings_data","sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","other"]} failure_code=0 failure_reason='

Any idea, whats causing this and how to fix it?

[thormanrd]

I getting the same thing from my indexers ever since I upgrade to v9.0.0.  We've had SmartStore running for a long time without these errors and now v9.0.0 has introduced this.  What is the timeout the the S2BacketCache is hitting?

11-18-2022 17:51:32.786 ERROR S2BucketCache [62311 BatchSearch] - waitFor, timed out trying to localize files from remote storage, cache_id="bid|att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340|" download_status=queued files={"file_types":["bloomfilter"]} local_files={"file_types":["sourcetypes_data","deletes"]} probe_count=148
11-18-2022 17:51:32.787 ERROR DatabaseDirectoryManager [62311 BatchSearch] - Bucket still not local after waiting, bid=att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340