SA-ldapsearch + TLS in a Search Head Cluster

[support0]

Hi there,

Does anyone here have succeeded in configuring SA-ldapsearch using TLS on a SHC ?

We have successfully configured it on a Heavy Forwarder part of our architecture but it does not work on a Search Head member of our Search Head Cluster where it does not seem to event load SSL settings.

Here is some details:

SSL Config is the same on both instances :

SH $ splunk cmd btool --app=SA-ldapsearch ssl list
[sslConfig]
caCertFile = /opt/splunk/etc/auth/ca.pem
sslVersions = tls

HF $ splunk cmd btool --app=SA-ldapsearch ssl list
[sslConfig]
caCertFile = /opt/splunk/etc/auth/ca.pem
sslVersions = tls

There is also an sslConfig stanza in another App, also indentical :

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
serverCert = $SPLUNK_HOME/etc/auth/splunk-cert.pem
requireClientCert = true
sslVerifyServerCert = true
certCreateScript =
sslVersions = tls1.1, tls1.2
sslVersionsForClient = tls1.1, tls1.2

SA-ldapsearch.log on the HF OK :

2017-09-11 18:56:32,218, Level=DEBUG, Pid=16868, File=search_command.py, Line=294, LdapTestConnectionCommand arguments: ['/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py', 'EXECUTE', 'domain="default"']
2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=search_command_internals.py, Line=296, LdapTestConnectionCommand: ldaptestconnection domain="default"
2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=ldaptestconnection.py, Line=48, Command = ldaptestconnection domain="default"
2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=configuration.py, Line=47, Command = ldaptestconnection domain="default"
2017-09-11 18:56:32,242, Level=DEBUG, Pid=16868, File=configuration.py, Line=536, Configuration = ldaptestconnection(server=[Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3), Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3), Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'*', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3)], credentials=CN=Splunk,OU=Admins,DC=domain,DC=local, alternatedomain=domain.local, basedn=dc=domain,dc=local, decode=True, paged_size=1000)
2017-09-11 18:56:32,242, Level=DEBUG, Pid=16868, File=ldaptestconnection.py, Line=65, Testing the connection to ldaps://domain.local:636

SA-ldapsearch.log on the SH KO, seems like SSL parameters are not loaded:

2017-09-11 18:54:06,998, Level=DEBUG, Pid=9063, File=search_command.py, Line=294, LdapTestConnectionCommand arguments: ['/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py', 'EXECUTE', 'domain="default"']
2017-09-11 18:54:07,000, Level=DEBUG, Pid=9063, File=search_command_internals.py, Line=296, LdapTestConnectionCommand: ldaptestconnection domain="default"
2017-09-11 18:54:07,000, Level=DEBUG, Pid=9063, File=ldaptestconnection.py, Line=48, Command = ldaptestconnection domain="default"
2017-09-11 18:54:07,001, Level=DEBUG, Pid=9063, File=configuration.py, Line=47, Command = ldaptestconnection domain="default"
2017-09-11 18:54:07,006, Level=ERROR, Pid=9063, File=search_command.py, Line=346, Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/search_command.py", line 320, in process
self.execute(operation, reader, writer)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/generating_command.py", line 79, in _execute
for record in operation():
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py", line 49, in generate
configuration = app.Configuration(self)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 52, in __init_
self.read_configuration()
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 432, in _read_configuration
settings = self._read_default_configuration()
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 457, in _read_default_configuration
response = service.get('properties/ldap/default', namespace.owner, namespace.app, namespace.sharing)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 241, in wrapper
return request_fun(self, *args, **kwargs)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 62, in new_f
val = f(*args, **kwargs)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 586, in get
response = self.http.get(path, self._auth_headers, **query)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1056, in get
return self.request(url, { 'method': "GET", 'headers': headers })
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1108, in request
response = self.handler(url, message, **kwargs)
File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1226, in request
connection.request(method, path, body, head)
File "/opt/splunk/lib/python2.7/httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "/opt/splunk/lib/python2.7/httplib.py", line 1097, in _send_request
self.endheaders(body)
File "/opt/splunk/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/opt/splunk/lib/python2.7/httplib.py", line 897, in _send_output
self.send(msg)
File "/opt/splunk/lib/python2.7/httplib.py", line 859, in send
self.connect()
File "/opt/splunk/lib/python2.7/httplib.py", line 1278, in connect
server_hostname=server_hostname)
File "/opt/splunk/lib/python2.7/ssl.py", line 352, in wrap_socket
_context=self)
File "/opt/splunk/lib/python2.7/ssl.py", line 579, in __init_
self.do_handshake()
File "/opt/splunk/lib/python2.7/ssl.py", line 808, in do_handshake
self._sslobj.do_handshake()
SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:603)

Thanks in advance for any feedback!

[matthewjohndavi]

Hi support0,

I currently have SA-ldapsearch 2.1.4 and TLS working in a SHC environment.

Try something like the below, and if it works for you, you can then harden it further.

This is in the SA-ldapsearch/local/ssl.conf file.

[sslConfig]
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
caCertFile = ca.pem
sslVersions = tls

Thanks,
Matt

[support0]

Hi Matt,

Thanks for your feedback.

We tried to play with sslVerifyServerCert & require Cert wihtout luck so far.

Quick question, you have mentioned SHC, do you execute ldad searches locally by setting :

[ldapsearch]
local = true

etc

or NOT ?

If not, I guess ldapsearch add-on is deployed on your Indexers as well, right ?

Thanks in advance,

[support0]

Hi Rafael,

Yes, Duane Waddle .conf are always consistent and were really helpful!

But in our case LDAP auth. is OK on the platform, what is not is ldapsearch and only on a SH part of the SHC.

[rafamss]

Hi @support0,

I found this doc and think that can help you to do a checklist in your configuration files: https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

In the page 14 he explain how to configurate the SH and in the page 24 the explanation to configurate UF (that can be used to HF).

In other document is explained how use security in Splunk: http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwith...

I hope help you.

[ ]s
Rafael Martins

[koshyk]

have you tried without ssl if it works on SHC?
Also, you mentioned about another app with sslConfig stanza. Can you please run a btool without the app=SA-ldapsearch? just to check if the other app is overriding the ldapsearch app

[support0]

Hi Koshyk,

Thanks for your help!

Indeed there is another App, but I meant to say that this App was deployed on both SH & HF. I might have been unclear.

If I run "splunk cmd btool ssl list", there is nothing interesting since ssl.conf is only used by ldapsearch while other apps use server.conf.

With "splunk cmd btool server list sslConfig --debug", there is one unique difference I did not spot earlier.

The SH checks for sslCommonNames while the HF does not:

sslCommonNameToCheck = splunk-searchhead,splunk-forwarder,splunk-indexer...

I did try to configure ldapsearch's ssl.conf with requireClientCert set to false and even with an empty sslCommonNameToCheck.

I did not try to put LDAP DC CommonName yet.

But I already went through sslCommonNameToCheck errors and I believe the generated error is pretty explicit in such case.

Might worth trying.

[koshyk]

I'm not a fan of SA-ldapsearch app as it is anyway too slow. I hope your LDAP is internal to the company?
What I would to make things moving is:
- Phase1 => Implement basic setup without certificates etc. See it working in a cluster properly. Let it work for a month 🙂
- Phase2 => Then try different settings which might break things.