Can we disable SSL between SH and peers?

[kwchang_splunk]

Dear experts?

When deploying cluster, can we disable SSL between peers and search head?
My customer's SH connects to peers over WAN, so the network bandwidth is not enough. All searches take several seconds before they display the results. Currently dispatch.createProviderQueue is 3~5 seconds.
I'd like to check whether disabling SSL can help this situation by skipping time for SSL handshaking.

Thank you.

[Drainy · Accepted solution]

Splunkd runs over SSL by default, so any communication will also be SSL. This isn't something you could disable just for that function as it would require splunkd to bind to another port.

Instead in server.conf you could disable SSL for Splunkd communication. This is probably a bad idea and personally in this case I would suggest that running this setup over a WAN isn't a great idea if you want to improve performance.

http://docs.splunk.com/Documentation/Splunk/5.0.3/admin/Serverconf

Other ways to spin it could be to setup local indexers and have the data forwarded over the WAN between them, not ideal and not the cheapest method but.. it could possibly improve performance at the expense of the speed at which data becomes available to search.

[kwchang_splunk]

SSL is turned off, but nodes are still trying to negotiate using PK. Can we turn off this behavior?

07-18-2013 10:35:44.191 +0900 ERROR NetUtils - Unable to negotiate ssl connection: error=1, Undefined error: 0
07-18-2013 10:35:44.191 +0900 ERROR NetUtils - SSL Error = error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
07-18-2013 10:35:44.191 +0900 WARN DistributedPeerManager - Send failure while pushing PK to search peer = https://xxx.xxx:8093, rv = 2 , http request to peer with uri=https://xxx.xxx:8093 returned an error. Check if the peer is up.

[kwchang_splunk]

That was the problem!
Thank you. Now it works.

[Drainy]

Presumably you've restarted them too? Did you also update any related URIs you've used for the license server, peer config etc to HTTP instead of HTTPS?

[kwchang_splunk]

I also disabled 'enableSplunkdSSL' in all peers. But it doesn't work.

[Drainy]

That sounds like enableSplunkdSSL is still enabled on the client?

[kwchang_splunk]

But.. there were so many errors in splunkd.log.
In master's splunkd.log :
07-15-2013 17:11:39.970 +0900 ERROR HTTPServer - Incomplete request="<80>V^A^C^A^@-^@^@^@ ...

In peers's splunkd.log :
07-15-2013 17:12:10.785 +0900 ERROR NetUtils - Unable to negotiate ssl connection: error=1, Undefined error: 0
07-15-2013 17:12:10.785 +0900 ERROR NetUtils - SSL Error = error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
07-15-2013 17:12:10.785 +0900 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
...

[Drainy]

Well I would expect it is the same setting on the client. Remember, this is just telling splunkd to use SSL or not, its regardless of its configuration as a server or client. Just the way it communicates, that would be my assumption anyway.

[kwchang_splunk]

Thank you for your comment.
I agree. I don't think this is a good idea. But I'm just trying this for testing. 🙂

I could turn off SSL of splunkd process using enableSplunkdSSL =false in server.conf
But I cannot find client side config for turning off SSL. I tried several parameters including useClientSSLCompression, useSplunkdClientSSLCompression but not successful so far.

Any idea?

Thank you in advance.