Hi, I am running a Search Head Cluster with 7 search heads on Splunk 8.2.9.
2 of the search heads are generating the following error messages at ~5 second intervals for a period of time before stopping:
ERROR DigestProcessor [38271 TcpChannelThread] - Failed signature match
ERROR HTTPAuthManager [38271 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/shcluster/member/consensus/pseudoid/raft_request_vote?output_mode=json
The search head cluster is otherwise running as expected as far as I can tell.
The search heads that are producing these errors are the only 2 that have been elected as captain in the last 30 days from examining the logs. There are no preferred captain or similar configurations set.
I have checked the [shclustering] pass4SymmKey values on each search head. They are all configured to the same value although use different Splunk Secrets to encrypt.
I am not sure when the errors first started appearing so can't link this to a specific upgrade on configuration change unfortunately.
The thread_id values seem to stay around for between 10-30 minutes. Sometimes 2 thread_ids will be active at once, sometimes none are active for a period. When looking at other logs for a particular thread_id around the same time period (at info logging level) I can't find see anything that adds any more cluses to what is causing the errors.
