Search Head Cluster: Failed HMAC signature match errors

a_kearney — Thu, 02 Feb 2023 10:32:05 GMT

Hi, I am running a Search Head Cluster with 7 search heads on Splunk 8.2.9.

2 of the search heads are generating the following error messages at ~5 second intervals for a period of time before stopping:

ERROR DigestProcessor [38271 TcpChannelThread] - Failed signature match ERROR HTTPAuthManager [38271 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/shcluster/member/consensus/pseudoid/raft_request_vote?output_mode=json

The search head cluster is otherwise running as expected as far as I can tell.

The search heads that are producing these errors are the only 2 that have been elected as captain in the last 30 days from examining the logs. There are no preferred captain or similar configurations set.

I have checked the [shclustering] pass4SymmKey values on each search head. They are all configured to the same value although use different Splunk Secrets to encrypt.

I am not sure when the errors first started appearing so can't link this to a specific upgrade on configuration change unfortunately.

The thread_id values seem to stay around for between 10-30 minutes. Sometimes 2 thread_ids will be active at once, sometimes none are active for a period. When looking at other logs for a particular thread_id around the same time period (at info logging level) I can't find see anything that adds any more cluses to what is causing the errors.

topic Search Head Cluster: Failed HMAC signature match errors in Splunk Enterprise

Search Head Cluster: Failed HMAC signature match errors