How to resolve Captain Connection error?

[Abdullah]

Dears,

We have the 3 SHC servers, and appearing below error in the health check icon on all SCH servers, you can see the screen shot.

Captain Connection

• Root Cause(s):
  • Classic bundle replication in Search Head Cluster has failed.
• Unhealthy Instances:

  • XXXXX

  Please your support for that.

  Best Regards,

[Abdullah]

Hi @richgalloway 

Only I can see below errors on splunkd.log for one of SHC.

10-31-2022 09:07:59.779 +0300 WARN ConfMetrics [173553 ConfReplicationThread] - single_action=PUSH_TO took wallclock_ms=28816! Consider a lower value of conf_replication_max_push_count in server.conf on all members

10-31-2022 09:08:15.562 +0300 WARN HttpListener [208689 HttpDedicatedIoThread-1] - Socket error from 127.0.0.1:49198 while accessing //servicesNS/nobody/splunk_secure_gateway/data/inputs/ssg_enable_modular_input/default/disable: Broken pipe
1

10-31-2022 09:08:20.433 +0300 WARN LocalAppsAdminHandler [100239 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.477 +0300 WARN LocalAppsAdminHandler [9534 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.513 +0300 WARN LocalAppsAdminHandler [34267 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.518 +0300 WARN LocalAppsAdminHandler [157525 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.558 +0300 WARN LocalAppsAdminHandler [165755 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.606 +0300 WARN LocalAppsAdminHandler [103480 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.609 +0300 WARN ConfReplicationThread [173553 ConfReplicationThread] - Error pushing configurations to captain=https://10.99.19.24:8089, consecutiveErrors=1 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=b171c01b88a8b8e9c8f9db9dbfcfb367de3ff073; current_baseline_op_id=a6fdcc3a9e93aa6d64c63fdc43005bc11210a972"
10-31-2022 09:08:20.617 +0300 WARN LocalAppsAdminHandler [34267 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.634 +0300 WARN LocalAppsAdminHandler [245085 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.640 +0300 WARN LocalAppsAdminHandler [34267 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.647 +0300 WARN LocalAppsAdminHandler [137654 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.852 +0300 WARN LocalAppsAdminHandler [100022 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.885 +0300 WARN LocalAppsAdminHandler [100022 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:20.955 +0300 WARN LocalAppsAdminHandler [7721 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.073 +0300 WARN LocalAppsAdminHandler [73456 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.077 +0300 WARN LocalAppsAdminHandler [127923 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.085 +0300 WARN LocalAppsAdminHandler [65584 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.280 +0300 WARN LocalAppsAdminHandler [179686 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.365 +0300 WARN LocalAppsAdminHandler [179686 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.368 +0300 WARN LocalAppsAdminHandler [99705 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.422 +0300 WARN LocalAppsAdminHandler [103480 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.430 +0300 INFO AutoLoadBalancedConnectionStrategy [199471 TcpOutEloop] - Connected to idx=10.99.19.32:9997:0, pset=0, reuse=0. autoBatch=1
10-31-2022 09:08:21.446 +0300 WARN LocalAppsAdminHandler [102067 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.451 +0300 WARN LocalAppsAdminHandler [100239 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.454 +0300 WARN LocalAppsAdminHandler [128022 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.459 +0300 WARN LocalAppsAdminHandler [103480 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.493 +0300 WARN LocalAppsAdminHandler [116746 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.512 +0300 WARN LocalAppsAdminHandler [116746 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
10-31-2022 09:08:21.545 +0300 WARN LocalAppsAdminHandler [116746 TcpChannelThread] - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf

Regards,

[richgalloway]

Have you checked splunkd.log?