@Treize On each SHC member, run: /opt/splunk/bin/splunk show shcluster-status -verbose The -verbose flag provides additional details, such as replication status, member health, and any pending ac...
See more...
@Treize On each SHC member, run: /opt/splunk/bin/splunk show shcluster-status -verbose The -verbose flag provides additional details, such as replication status, member health, and any pending actions. Look for discrepancies (e.g., a member marked as “Pending” or “Out of Sync”). Check the splunkd.log on all four SHC members for errors related to clustering. Focus on: SHCMaster, SHCMember, or ConfReplication components. Errors like “failed to proxy call” or “replication failure.” The GUI might be stuck due to a caching issue. Restart Splunk Web (not the full splunkd process) on all members: /opt/splunk/bin/splunk restart If the GUI persists in showing incorrect data, resynchronize the replicated configuration across the cluster: On the captain, run: /opt/splunk/bin/splunk resync shcluster-replicated-config https://docs.splunk.com/Documentation/Splunk/9.2.1/DistSearch/HowconfrepoworksinSHC . This command pulls the latest configuration from the captain and pushes it to all members, which might realign the GUI’s view. Rolling Restart (If Needed): If the above steps don’t resolve the issue, perform a controlled rolling restart of the cluster: /opt/splunk/bin/splunk rolling-restart shcluster-members This ensures all members restart cleanly and re-register with the captain, potentially fixing any GUI misalignment. Monitor the GUI and CLI status post-restart. Splunk 7.3.5 is several years old, and while it’s stable for many environments, there have been reported bugs in SHC management and GUI rendering in earlier 7.x versions. Check Splunk’s Known Issues documentation https://docs.splunk.com/Documentation/Splunk/7.3.5/ReleaseNotes/KnownIssues