Error in removing insecure TLS cipher suites in indexpeer replication

[WurschtHans]

Hi,

I want to remove insecure tls cipher suites from indexpeer replication.

The default setting in server.conf/[sslConfig] is:

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

However, if I remove the insecure ciphers

AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

From cipherSuite and deploy that configuration to our indexpeers, indexpeer replication won't work anymore.

splunkd.log of one of our indexpeers after the configuration change:

06-09-2020 13:41:08.732 +0200 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv2/v3 read server hello A', alert_description='handshake failure'.
06-09-2020 13:41:08.732 +0200 ERROR TcpOutputFd - Connection to host=10.10.10.10:9101 failed. sock_error = 0. SSL Error = error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
06-09-2020 13:41:08.733 +0200 WARN BucketReplicator - Connection failed

We are using Splunk 8.0.4.

Has anyone succeeded in securing Splunk?

Thanks!

[harsmarvania57]

Are you using below config for replication of buckets ?

[replication_port-ssl://<port>]
* This configuration is the same as the replication_port stanza, but uses SSL.

[WurschtHans]

Yes, we are using 

[replication_port-ssl://9101]

[harsmarvania57]

Sorry for late reply, how are you deploying configuration to Indexers ? Have you changed cipherSuite on all Indexers and restarted splunk on Indexers ? Can you please check output of below command on all Indexers, is it same ? If yes then can you please provide output from any one Indexer ?

$SPLUNK_HOME/bin/splunk show config servers | grep -i cipher

[WurschtHans]

We changed cipherSuites on all our indexpeers. The configuration (server.conf) was deployed to the indexpeers directly via ansible.

Afterwards we issued a cluster-restart from from the clustermaster (splunk rolling-restart cluster-peers).

Here is the the output after the restart:

>> /data/splunk/install/splunk/bin/splunk show config server | grep cipher
Your session is invalid.  Please login.
Splunk username: xyyxcvyxcv
Password:
cipherSuite=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
cipherSuite=TLSv1.2+HIGH:@STRENGTH
cipherSuite=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256

[harsmarvania57]

From the output I can see that below cipher does not exist on your indexer. Are you still receiving replication error ?

AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

[WurschtHans]

Yes. I suspect, that the used ciphers are somehow hardcoded to that insecure versions.

Can you reproduce this error?

[harsmarvania57]

Sorry I don't have environment setup with SSL Replication on Indexer Cluster so can't test it.