Splunk Enterprise

Search Head Cluster: Failed HMAC signature match errors


Hi, I am running a Search Head Cluster with 7 search heads on Splunk 8.2.9.

2 of the search heads are generating the following error messages at ~5 second intervals for a period of time before stopping:



ERROR DigestProcessor [38271 TcpChannelThread] - Failed signature match
ERROR HTTPAuthManager [38271 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/shcluster/member/consensus/pseudoid/raft_request_vote?output_mode=json



The search head cluster is otherwise running as expected as far as I can tell.

The search heads that are producing these errors are the only 2 that have been elected as captain in the last 30 days from examining the logs. There are no preferred captain or similar configurations set.

I have checked the [shclustering] pass4SymmKey values on each search head. They are all configured to the same value although use different Splunk Secrets to encrypt.

I am not sure when the errors first started appearing so can't link this to a specific upgrade on configuration change unfortunately.

The thread_id values seem to stay around for between 10-30 minutes. Sometimes 2 thread_ids will be active at once, sometimes none are active for a period. When looking at other logs for a particular thread_id around the same time period (at info logging level) I can't find see anything that adds any more cluses to what is causing the errors.


Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...

Platform Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestIntroducing Splunk Edge Processor, simplified data ...