Poor ldapsearch performance - SA-ldapsearch app

[Lucas_K]

Has anyone figured out the root cause of the ridiculously poor performance of the SA-ldapsearch apps ldapsearch command?

I've hunted through the code, i've wiresharked the communications and the delays are all internally within the app itself.
I've tried active directory servers (millions of entries, redirections etc), openldap (~10 groups, 40 users) and both respond in the same time frame with the same splunk-side delays.

I've tried ssl and non-ssl connections and the slow queries remain.

There is a minimum 10 second delay before the query is sent to the ldap server.

You can see it in the picture below.

[rkantamaneni_sp]

Hi All, On Jan. 3rd, 2019, Splunk released v2.20 of SA-ldapsearch. This has the fix for the performance issues by integration of newer libraries. You can see the notes in "What's New" of the Release Notes:

https://docs.splunk.com/Documentation/SA-LdapSearch/2.2.0/User/ReleaseNotes

Hopefully this should resolve any issues with performance people were facing, already heard good feedback from others.

[RupeshMano]

after applying the patch, ldapsearch was working fine and but I had issues with ldapgroup

ERROR

External search command 'ldapgroup' returned error code 1. First 1000 (of 3858669) bytes of script output: "error_message=AttributeError at "/opt/splunk/etc/apps/SA-ldapsearch/bin/ldapgroup.py", line 130 : 'module' object has no attribute 'LDAPException' serial,mv_serial,time,mv_time,raw,mvraw,host,mv_host,dn,mv_dn,distinguishedName,mv_distinguishedName,errors,mv_errors,member_dn,mv_member_dn,member_domain,mv_member_domain,member_name,mv_member_name,member_type,mv_member_type,mv_combo,_mv_mv_combo 0,,1533622287.24,,"

I had to do the following to overcome this error.

1. Replace ldap3.* as ldap3.core.exceptions.* in ldapgroup.py for all the exceptions.
   Example:
   except ldap3.LDAPNoSuchObjectResult:
   to
   except ldap3.core.exceptions.LDAPNoSuchObjectResult:

2. Add communication in $SPLUNK_HOME/etc/apps/SA-ldapsearch/bin/packages/ldap3/__init.py (This was there in the package which ships with SA-ldapsearch but not in the new ldap3 version)
   "

   communication

   SESSION_TERMINATED_BY_SERVER = 0
   
   RESPONSE_COMPLETE = -1
   
   RESPONSE_SLEEPTIME = 0.02 # seconds to wait while waiti
   
   RESPONSE_WAITING_TIMEOUT = 10 # waiting timeout for rec
   
   SOCKET_SIZE = 4096 # socket byte size
   
   "
   Making the above changes had good results on ldapgroup command, but still there are python errors sometimes due to the huge extraction time.

[datasearchninja · Accepted solution]

I have had really good results by upgrading the ldap3 library that ships with the package to the latest version. This is complicated by a change in name of some attributes, and a dependancy on ctypes which also makes it non portable. However I'm seeing a reduction in time of a full query of one directory from 4 hours to 3 minutes.

https://github.com/cannatag/ldap3/archive/v2.5.tar.gz
http://downloads.sourceforge.net/project/ctypes/ctypes/1.0.2/ctypes-1.0.2.tar.gz

On a system with python + python-devel 2.7 (RHEL 7):

$ cd ~
$ tar zxf ctypes-1.0.2.tar.gz
$ cd ctypes-1.0.2
$ python setup.py build
$ tar zxf ldap3-2.5.tar.gz
$ cd ldap3-2.5
$ python setup.py build
$ cd /opt/splunk/etc/apps/
$ tar zxvf ~/splunk-supporting-add-on-for-active-directory_217.tgz
$ rm -fR SA-ldapsearch/bin/packages/ldap3
$ mv ~/ctypes-1.0.2/build/lib.linux-x86_64-2.7/* SA-ldapsearch/bin/packages/
$ mv ~/ldap3-2.5/build/lib/ldap3 SA-ldapsearch/bin/packages/
$ cd SA-ldapsearch
$ patch -p1 <  ~/SA-ldapsearch.patch

The patch

$ cat ~/SA-ldapsearch.patch
diff -Naur ./SA-ldapsearch.2.1.7/bin/ldapfetch.py SA-ldapsearch/bin/ldapfetch.py
--- ./SA-ldapsearch.2.1.7/bin/ldapfetch.py      2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/ldapfetch.py      2018-06-25 14:27:13.465247255 +1000
@@ -64,7 +64,7 @@
         """
         configuration = app.Configuration(self, is_expanded=True)
         expanded_domain = app.ExpandedString(self.domain)
-        search_scope = ldap3.SEARCH_SCOPE_BASE_OBJECT
+        search_scope = ldap3.BASE
         search_filter = '(objectClass=*)'

         try:
diff -Naur ./SA-ldapsearch.2.1.7/bin/ldapfilter.py SA-ldapsearch/bin/ldapfilter.py
--- ./SA-ldapsearch.2.1.7/bin/ldapfilter.py     2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/ldapfilter.py     2018-06-25 14:27:13.465247255 +1000
@@ -50,9 +50,9 @@
         **Default:** sub.
         ''',
         default='sub', validate=validators.Map(
-            base=ldap3.SEARCH_SCOPE_BASE_OBJECT,
-            one=ldap3.SEARCH_SCOPE_SINGLE_LEVEL,
-            sub=ldap3.SEARCH_SCOPE_WHOLE_SUBTREE
+            base=ldap3.BASE,
+            one=ldap3.LEVEL,
+            sub=ldap3.SUBTREE
         ))

     decode = Option(
diff -Naur ./SA-ldapsearch.2.1.7/bin/ldapgroup.py SA-ldapsearch/bin/ldapgroup.py
--- ./SA-ldapsearch.2.1.7/bin/ldapgroup.py      2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/ldapgroup.py      2018-06-25 14:27:13.465247255 +1000
@@ -68,7 +68,7 @@
         """
         configuration = app.Configuration(self, is_expanded=True)
         expanded_domain = app.ExpandedString(self.domain)
-        search_scope = ldap3.SEARCH_SCOPE_BASE_OBJECT
+        search_scope = ldap3.BASE
         search_filter = '(objectCategory=Group)'
         attributes = ['objectSid']

diff -Naur ./SA-ldapsearch.2.1.7/bin/ldapsearch.py SA-ldapsearch/bin/ldapsearch.py
--- ./SA-ldapsearch.2.1.7/bin/ldapsearch.py     2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/ldapsearch.py     2018-06-25 14:27:13.465247255 +1000
@@ -14,6 +14,7 @@
 from time import time
 import ldap3
 import app
+import datetime


 @Configuration(retainsevents=True)
@@ -53,9 +54,9 @@
         **Default:** sub.
         ''',
         default='sub', validate=validators.Map(
-            base=ldap3.SEARCH_SCOPE_BASE_OBJECT,
-            one=ldap3.SEARCH_SCOPE_SINGLE_LEVEL,
-            sub=ldap3.SEARCH_SCOPE_WHOLE_SUBTREE
+            base=ldap3.BASE,
+            one=ldap3.LEVEL,
+            sub=ldap3.SUBTREE
         ))

     debug = Option(
@@ -108,13 +109,14 @@
                         yield LdapSearchCommand._record(
                             serial_number, time_stamp, connection.server.host, dn, attributes, attribute_names, encoder)
                         serial_number += 1
+                        GeneratingCommand.flush
                     if self.limit and serial_number == self.limit:
                         break
                     pass

                 pass

-        except ldap3.LDAPException as error:
+        except ldap3.core.exceptions.LDAPException as error:
             self.error_exit(error, app.get_ldap_error_message(error, configuration))

         return
@@ -127,10 +129,14 @@
         for name, value in attributes.iteritems():
             if isinstance(value, str):
                 attributes[name] = b64encode(value)
+            elif isinstance(value, datetime.datetime):
+                attributes[name] = str(value)
             elif isinstance(value, list):
                 for i in range(len(value)):
                     if isinstance(value[i], str):
                         value[i] = b64encode(value[i])
+                    elif isinstance(value[i], datetime.datetime):
+                        value[i] = str(value[i])

         raw = encoder.encode(attributes)

diff -Naur ./SA-ldapsearch.2.1.7/bin/ldaptestconnection.py SA-ldapsearch/bin/ldaptestconnection.py
--- ./SA-ldapsearch.2.1.7/bin/ldaptestconnection.py     2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/ldaptestconnection.py     2018-06-25 14:27:13.465247255 +1000
@@ -11,7 +11,7 @@
 from collections import Iterable
 from itertools import chain
 from json import JSONEncoder
-from ldap3 import Connection, SEARCH_SCOPE_BASE_OBJECT
+from ldap3 import Connection, BASE
 from ldap3.core.exceptions import LDAPException
 from ldapsearch import LdapSearchCommand
 from time import time
@@ -55,7 +55,7 @@
         servers = configuration.server if isinstance(configuration.server, Iterable) else (configuration.server,)
         search_base = configuration.basedn
         search_filter = '(objectClass=*)'
-        search_scope = SEARCH_SCOPE_BASE_OBJECT
+        search_scope = BASE
         attribute_names = 'distinguishedName',

         records = []
diff -Naur ./SA-ldapsearch.2.1.7/bin/packages/app/configuration.py SA-ldapsearch/bin/packages/app/configuration.py
--- ./SA-ldapsearch.2.1.7/bin/packages/app/configuration.py     2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/packages/app/configuration.py     2018-06-25 14:25:40.464586343 +1000
@@ -9,7 +9,7 @@
 import ssl
 import sys

-from ldap3 import Server, Tls, LDAPSSLConfigurationError, GET_ALL_INFO
+from ldap3 import Server, Tls, core, ALL
 from splunklib.searchcommands.validators import Boolean, Integer, List, Map
 from splunklib.binding import HTTPError
 from splunklib import data
@@ -318,10 +318,8 @@
             tls = Tls(
                 ca_certs_file=ca_cert_file if ca_cert_file else None,
                 validate=ssl.CERT_REQUIRED if ssl_verify_server_cert else ssl.CERT_NONE,
-                version=version,
-                use_ssl_context=True if version==ssl.PROTOCOL_SSLv23 else False,
-                ssl_no_v2=ssl_no_v2, ssl_no_v3=ssl_no_v3)
-        except LDAPSSLConfigurationError as error:
+                version=version)
+        except core.exceptions.LDAPSSLConfigurationError as error:
             message = 'SSL configuration issue: {0}'.format(error)
             command.error_exit(error, message)
             return
@@ -365,7 +363,7 @@
                 ca_certs_file=ca_cert_file if ca_cert_file else None,
                 validate=ssl.CERT_REQUIRED if ssl_verify_server_cert else ssl.CERT_NONE,
                 version=version)
-        except LDAPSSLConfigurationError as error:
+        except core.exceptions.LDAPSSLConfigurationError as error:
             message = 'SSL configuration issue: {0}'.format(error)
             command.error_exit(error, message)
             return
@@ -527,7 +525,7 @@

         def create_server(hostname):
             return Server(
-                hostname, int(port), use_ssl, formatter=formatter, get_info=GET_ALL_INFO,
+                hostname, int(port), use_ssl, formatter=formatter, get_info=ALL,
                 allowed_referral_hosts=[('*', True)], tls=tls)

         self.server = create_server(host[0]) if len(host) == 1 else [create_server(h) for h in host]
diff -Naur ./SA-ldapsearch.2.1.7/bin/packages/app/__init__.py SA-ldapsearch/bin/packages/app/__init__.py
--- ./SA-ldapsearch.2.1.7/bin/packages/app/__init__.py  2018-05-29 22:53:10.000000000 +1000
+++ SA-ldapsearch/bin/packages/app/__init__.py  2018-06-25 14:25:40.463586336 +1000
@@ -324,7 +324,7 @@
     # Example: The message produced for LDAPInvalidCredentialsResult
     error.message = error.message.replace('\0', '')

-    if not isinstance(error, ldap3.LDAPInvalidCredentialsResult):
+    if not isinstance(error, ldap3.core.exceptions.LDAPInvalidCredentialsResult):
         return error.message

     try:
@@ -384,12 +384,12 @@
     is_microsoft_active_directory = True

     for feature in supported_features:
-        if feature.oid == '1.3.6.1.4.1.4203.1.5.1':  # This DSA supports "All Operational Attributes" as per RFC 3673
+        if feature[0] == '1.3.6.1.4.1.4203.1.5.1':  # This DSA supports "All Operational Attributes" as per RFC 3673
             supports_all_operational_attributes = True
             break
-        if feature.docs != 'MICROSOFT':
+        if feature[3] != 'MICROSOFT':
             is_microsoft_active_directory = False
-        if feature.oid in _required_features:
+        if feature[0] in _required_features:
             required_features_countdown -= 1

     if not (is_microsoft_active_directory and required_features_countdown == 0):

[koshyk]

will try it. thanks for the update

[datasearchninja]

As an update, I've seen the ctypes stuff is only used in the directory sync portion of the ldap3 library, so removing that bit means the ctypes library isn't needed, and the app can stay portable...

Just comment out these 2 lines:

--- ldap3/protocol/microsoft.py.orig    2018-07-04 09:02:54.502926488 +1000
+++ ldap3/protocol/microsoft.py 2018-07-03 20:58:38.366085375 +1000
@@ -23,7 +23,7 @@
 # along with ldap3 in the COPYING and COPYING.LESSER files.
 # If not, see <http://www.gnu.org/licenses/>.

-import ctypes
+#import ctypes

 from pyasn1.type.namedtype import NamedTypes, NamedType
 from pyasn1.type.tag import Tag, tagClassApplication, tagFormatConstructed
@@ -112,7 +112,7 @@
         flags |= 0x80000000
         # converts flags to signed 32 bit (AD expects a 4 bytes long unsigned integer, but ASN.1 Integer type is signed
         # so the BER encoder gives back a 5 bytes long signed integer
-        flags = ctypes.c_long(flags & 0xFFFFFFFF).value
+        #flags = ctypes.c_long(flags & 0xFFFFFFFF).value
 control_value.setComponentByName('Flags', flags)
 control_value.setComponentByName('MaxBytes', max_length)

[RupeshMano]

after applying the patch, ldapsearch is working fine and I am having issues with ldapgroup, does anyone faces same issue ?

Running the below query I am getting error

| ldapsearch search="(objectClass=group)" attrs=distinguishedName
| ldapgroup

ERROR

External search command 'ldapgroup' returned error code 1. First 1000 (of 3858669) bytes of script output: "error_message=AttributeError at "/opt/splunk/etc/apps/SA-ldapsearch/bin/ldapgroup.py", line 130 : 'module' object has no attribute 'LDAPException' serial,mvserial,_time,mvtime,_raw,mvraw,host,mv_host,dn,mv_dn,distinguishedName,mv_distinguishedName,errors,mv_errors,member_dn,mv_member_dn,member_domain,mv_member_domain,member_name,mv_member_name,member_type,mv_member_type,mv_combo,_mv_mv_combo 0,,1533622287.24,,"

[dfronck]

I tested this without the ctypes part and one of our daily asset queries is back to one minute instead of +30 minutes. Thanks Colin!

[Lucas_K]

This works great thanks!

I really appreciate the time you spent to get this working.

This actually makes all the ldap dependent dashboards in the windows app work now 🙂

[Richfez]

While I'm sure you have seen this, but for others who may stumble across this question looking for a solution to a less intractable version of this problem, the following tip may help a lot:

How to speed up LDAP

[Lucas_K]

I've actually found the root cause with is a module that is included with this app.

As the source is no longer available there is noupdated method to utilise to fix this.

In a nutshell there is a bug in the underlying dependencies that causes super slow performance in any ldap environment larger than a few users.

I tried it against an internal group ldap server which only had a few groups and members and it was fine.

[marklaw2]

I finally gave up on " | ldapsearch ". Since our LDAP environment doesn't change all that rapidly, it was easier to write a Perl script to pull all the information nightly in a cron job and dump it to a *.csv file. Lookups using the "|inputlookup my_domain.csv ", with a properly formatted CSV file can be used in most places that call for the ldapsearch construct.

[twinspop]

any chance you can share a properly redacted version?

[dfronck]

This is a known issue in v2 and has been for 2 years. Can't see it ever getting fix.

2015-12-4 TAG-9567 After you upgrade from version 1.x to version 2.x of the add-on, the custom commands (ldapsearch,
ldapfetch, ldapfilter, ldapgroup, ldaptestconnection) are slower.

The only "solution" is to roll back to v1. (Not a good solution.)

[koshyk]

same here. Very poor performance.from SA-ldapsearch. It takes 4 hours to get identities from clients AD system using the app. When I use the openldap tool, it fetches in 7 minutes.
I've written a script to collect data using openldap tool and extract into a csv. Splunk will then use this lookup as an external lookup. I'm planning to make this into an app hopefully when i get time.

[peterchenadded]

Are you guys on windows? With slow disk it can take some time to load all the libraries, but haven't seem it take that long.

[koshyk]

we are 100% on linux. the identity list is quite huge, but still I cannot justify the saved search to fetch AD takes 4 hrs. I think I've raised it as a support case quite long before with others too raised.

[peterchenadded]

Are you running a search head cluster?

If so is the command running locally on the search head or across all indexers?

[Lucas_K]

Standalone splunk search head with distributed search utilising local = true on all commands.conf stanza's.

This means it is only running on the search head.