Hi, I am running a Search Head Cluster with 7 search heads on Splunk 8.2.9. 2 of the search heads are generating the following error messages at ~5 second intervals for a period of time before stopping: ERROR DigestProcessor [38271 TcpChannelThread] - Failed signature match
ERROR HTTPAuthManager [38271 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/shcluster/member/consensus/pseudoid/raft_request_vote?output_mode=json The search head cluster is otherwise running as expected as far as I can tell. The search heads that are producing these errors are the only 2 that have been elected as captain in the last 30 days from examining the logs. There are no preferred captain or similar configurations set. I have checked the [shclustering] pass4SymmKey values on each search head. They are all configured to the same value although use different Splunk Secrets to encrypt. I am not sure when the errors first started appearing so can't link this to a specific upgrade on configuration change unfortunately. The thread_id values seem to stay around for between 10-30 minutes. Sometimes 2 thread_ids will be active at once, sometimes none are active for a period. When looking at other logs for a particular thread_id around the same time period (at info logging level) I can't find see anything that adds any more cluses to what is causing the errors.
... View more