Deployment Architecture

Why am I getting partial results on some indexers?

kyaparla
Path Finder

Seeing ERROR message "may have returned partial results" from few indexers".  Logs from those indexers are showing following error messages.

 

WARN CacheManagerHandler - Localization failure has been reported, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", sid="remote_*_1646235912.747477_6AFBB424-8451-40A4-A05C-A0337BDBC296", errorMessage='waitFor probe, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", did not localize all files before reaching download_status=idle files={"file_types":["tsidx","bloomfilter","deletes"]} local_files={"file_types":["dma_metadata","strings_data","sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","other"]} failure_code=0 failure_reason='

 

Any idea, whats causing this and how to fix it?

Labels (1)
0 Karma

thormanrd
Path Finder

I getting the same thing from my indexers ever since I upgrade to v9.0.0.  We've had SmartStore running for a long time without these errors and now v9.0.0 has introduced this.  What is the timeout the the S2BacketCache is hitting?

11-18-2022 17:51:32.786 ERROR S2BucketCache [62311 BatchSearch] - waitFor, timed out trying to localize files from remote storage, cache_id="bid|att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340|" download_status=queued files={"file_types":["bloomfilter"]} local_files={"file_types":["sourcetypes_data","deletes"]} probe_count=148
11-18-2022 17:51:32.787 ERROR DatabaseDirectoryManager [62311 BatchSearch] - Bucket still not local after waiting, bid=att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...