Why is shc captain flapping- constant election going on every several minutes?

[kundeng]

index=_internal sourcetype=splunkd NOT info (log_level="WARN" OR log_level="ERROR" OR log_level="CRIT" OR log_level="FATAL") splunksh* component=SHCMasterHTTPProxy log_level=WARN

06-08-2021 17:11:11.735 -0400 WARN SHCMasterHTTPProxy - Low Level http request failure err=failed method=POST path=/services/shcluster/captain/artifacts/dummy/remove_all captain=splunkshc-prod05.med.umich.edu:8089 rc=0 actual_response_code=500 expected_response_code=200 status_line="Internal Server Error" transaction_error="<response>\n <messages>\n <msg type="ERROR">Peer 3AB94F34-364E-450C-81AC-D6E4A2E6018A does not exist. Cannot remove artifacts/sids from all</msg>\n </messages>\n</response>\n"

It happens on all nodes, but more frequent when 2 out of 5 are captains:

xxa.med.umich.edu:8089 52
xxb.med.umich.edu:8089 3253
xxc.med.umich.edu:8089 45
xxd.med.umich.edu:8089 107
xxe.med.umich.edu:8089 3799

[erw550]

Hello,

Any updates regarding this? We have the same problem.

[marcia01]

Hello, @kundeng

Did you resolve this probleam? We have the same symptoms in our environment at version 8.1.3.
Thanks