Deployment Architecture

SHClustering SSLv3 Errors

Path Finder

Hey guys,

Been troubleshooting my distributed searching for the last two days. My environment:

Primary facility:
Cluster Master
deployment server
2 Search Heads
2 indexers

2 Search Heads ( captain is here for testing)
2 indexers

the 2 search headers in Primary can't reach the captain and splunkd.log is showing the following:

WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSL negotiation finished successfully', alert_description='bad record mac'.
08-14-2019 18:04:48.986 +0000 ERROR HttpClientRequest - HTTP client error=error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac while accessing server=https://:8089 for request=https://:8089/services/shcluster/captain/members.

On the cluster master i'm seeing the following:
(obfuscating the IPs)
Bundle Replication: Problem replicating config (bundle) to search peer ' secondary indexer 1 ', Unknown write error
Bundle Replication: Problem replicating config (bundle) to search peer ' secondary indexer 2 ', Unknown write error

so it seems to me that what ever Master ( search or clustermaster) in either site, can't talk to devices in the other site.

few thoughts:
There is a F5 in between
other is it's going over a WAN and latency is effecting it.

Anyone have ideas?

0 Karma


This alert is returned if a record is received with an incorrect MAC. This alert also MUST be returned if an alert is sent because a TLSCiphertext decrypted in an invalid way: either it wasn't an even multiple of the block length, or its padding values, when checked, weren't correct.

Possible causes:
SNAT isn't enabled
F5 is unloading ssl and rebaking with F5's cert
Wrong ciphers in use on both ends

Check server.conf [ssl] & [shclustering] stanzas on each search head.

Also you mentioned one search head is captain. Are you running static captain? If so that's only supported for when you're recovering a failed cluster.

0 Karma


You have an F5 between sites? Why?

If this reply helps you, Karma would be appreciated.
0 Karma

Path Finder

this is because this is two separate subnets. We use the F5 as a gateway to route the traffic

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...