Deployment Architecture

Why am I getting partial results on some indexers?

kyaparla
Path Finder

Seeing ERROR message "may have returned partial results" from few indexers".  Logs from those indexers are showing following error messages.

 

WARN CacheManagerHandler - Localization failure has been reported, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", sid="remote_*_1646235912.747477_6AFBB424-8451-40A4-A05C-A0337BDBC296", errorMessage='waitFor probe, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", did not localize all files before reaching download_status=idle files={"file_types":["tsidx","bloomfilter","deletes"]} local_files={"file_types":["dma_metadata","strings_data","sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","other"]} failure_code=0 failure_reason='

 

Any idea, whats causing this and how to fix it?

Labels (1)
0 Karma

thormanrd
Path Finder

I getting the same thing from my indexers ever since I upgrade to v9.0.0.  We've had SmartStore running for a long time without these errors and now v9.0.0 has introduced this.  What is the timeout the the S2BacketCache is hitting?

11-18-2022 17:51:32.786 ERROR S2BucketCache [62311 BatchSearch] - waitFor, timed out trying to localize files from remote storage, cache_id="bid|att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340|" download_status=queued files={"file_types":["bloomfilter"]} local_files={"file_types":["sourcetypes_data","deletes"]} probe_count=148
11-18-2022 17:51:32.787 ERROR DatabaseDirectoryManager [62311 BatchSearch] - Bucket still not local after waiting, bid=att_ent_flows~42619~99400A76-ADCE-4D97-A8E9-9DF39952E340

0 Karma
Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials (Episode 3)

Welcome back to Splunk Classroom Chronicles, our ongoing blog series that pulls back the curtain on Splunk ...

Operationalizing TDIR: Building a More Resilient, Scalable SOC

Optimizing SOC workflows with a unified, risk-based approach to Threat Detection, Investigation, and Response ...

Almost Too Eventful Assurance: Part 1

Modern IT and Network teams still struggle with too many alerts and isolating issues before they are notified. ...