Why is the replication status on distributed search failing with this error "Gave up waiting for the captain to establish a common bundle version across all search peers"?

[splunker9999]

Hi, We are seeing replication status on search head captain and one of the search head has failed.

Also we are seeing below error when we tried to search any data.

Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead

We tried restarting search head cluster, rolling restart and resync, but still facing this issue.

Can you please advice how we can handle this issue?

Thanks.

[sombhtr239]

Hello,

None of the above statement hold true for me. Our VM's are hosted in GCP, and we have 3 SH member in the SHC and 3 IDX. The issue is observed only in 1 peer, while the other works fine. Tried the setting mentioned above. Also, removed and reconfigured the problematic SH. The issue persists. If anyone has resolved the issue, please suggest steps to resolve this issue.

[bmacias84]

I've seen this in two cases.

1. Your search heads don't have enough resources. in server.conf consider setting captain_is_adhoc_searchhead = True
2. The other case would be in your bundles are over 800MB which is the default max http content size which most likely are very large lookup table(s). The DMC should be able to tell you how large your bundles are. Delete the large items or consider increasing max_content_length which located in server.conf.