topic Re: How do I improve ldapsearch performance in Security

How do I improve ldapsearch performance

napomokoetle — Thu, 26 Nov 2015 11:54:50 GMT

My environment:

Splunk Supporting Add-on for Active Directory 2.1.1
Splunk Enterprise 6.3.1
Running on Linux Centos 6.5

When I execute any LDAP search I have to wait for at least 5 minutes before I see results back! This is extremely slow. What can I do to troubleshoot this performance problem and improve the performance?

Kind regards,
J. Napo Mokoetle

Re: How do I improve ldapsearch performance

Lucas_K — Thu, 26 Nov 2015 13:36:01 GMT

This has been broken for as long as it has existed.

I check with an os based ldapsearch and it instantly returns. As soon as you try to do it in the addon it's pretty much useless. We have a massive number of use cases for this and we have to resort to using external tools 😞

Re: How do I improve ldapsearch performance

napomokoetle — Thu, 26 Nov 2015 15:43:52 GMT

Frustrating! Let's hope someone from Splunk or otherwise will hear our cry and help us out of our specific ldapsearch related pain Lucas K 😉 .

Re: How do I improve ldapsearch performance

MuS — Sun, 29 Nov 2015 19:32:28 GMT

Hi napomokoetle,

just a quick question: do you use the LDPA Add-on or the SA-ldapsearch? I'm asking because you tagged the question with Add-on for LDAP which does not provide a ldapsearch command and listed the Add-on for Active Directory as well...

cheers, MuS

Re: How do I improve ldapsearch performance

napomokoetle — Mon, 30 Nov 2015 06:37:00 GMT

Hi MuS, Thanks for your response. I'm using the SA-ldapsearch. I hope you can help resolve this performance problem.
Just to check if Splunk was the problem or NOT. I installed the Centos ldapsearch client and executed the same Active Directory search I've been trying through Splunk, and the AD results return in seconds.
So it seems to me the problem is with Splunk and not Active Directly or my system.

Re: How do I improve ldapsearch performance

gwalford — Thu, 11 Feb 2016 18:35:03 GMT

I am dealing with a large domain, and an LDAPsearch takes over 8 hours to complete - but it usually times out.

Re: How do I improve ldapsearch performance

napomokoetle — Thu, 11 Feb 2016 20:34:45 GMT

I suppose Splunk also has no clue on fixing this one or helping us get it right.

Re: How do I improve ldapsearch performance

MuS — Thu, 11 Feb 2016 20:39:27 GMT

Hi napomokoetle,

did you open a support case?

Re: How do I improve ldapsearch performance

napomokoetle — Fri, 12 Feb 2016 03:27:29 GMT

Hi Mus,

I have had the problem for a very long time and don't recall opening a trouble ticket. But there are many others who have had the same problem and opened tickets but I believe they never had the problem solved.

Re: How do I improve ldapsearch performance

rtestu_splunk — Wed, 16 Mar 2016 19:52:07 GMT

Hello,

Have you tried this :

To improve performance on queries against ADs with large numbers of users, select only the query attributes you need to complete your analysis. For example, if you need just two attributes, distinguishedName and sAMAccountName, say so. Use this command:

| ldapsearch search="(objectClass=user)" attrs="distinguishedName,sAMAccountName"

instead of:

| ldapsearch search="(objectClass=user)"

It comes from the documentation :
http://docs.splunk.com/Documentation/SA-LdapSearch/latest/User/UseSA-ldapsearchtotroubleshootproblems

I tried and it significantly improved our performance.

Hope it will help !
Romain.

Re: How do I improve ldapsearch performance

napomokoetle — Thu, 17 Mar 2016 12:45:13 GMT

Hi rtestu,

Thanks for the response.
I do use very specific searches. But the result is still very sluggish response. Still have to weight at last 5 minutes for any results to show up!
Here's an example query I execute:

search="(&(objectclass=user)(!(objectClass=computer))(accountExpires=9223372036854775807))"|sort sAMAccountName|table sAMAccountName,cn,userPrincipalName,userAccountControl

Could someone perhaps suggest ways to trace the situation to get to where the bottleneck could be?

Re: How do I improve ldapsearch performance

rtestu_splunk — Thu, 17 Mar 2016 13:04:31 GMT

And did you try what I mentioned ?

| ldapsearch search="(&(objectclass=user)(!(objectClass=computer))(accountExpires=9223372036854775807))"
attrs="sAMAccountName,cn,userPrincipalName,userAccountControl"
| sort sAMAccountName
| table sAMAccountName,cn,userPrincipalName,userAccountControl

It could significantly improve the performance if you have to retrieve lots of data from the LDAP. In your case, I am not sure it will improve a lot since you seem to retrieve a few accounts ...

Romain.

Re: How do I improve ldapsearch performance

napomokoetle — Thu, 17 Mar 2016 13:42:07 GMT

Hi retestu,

Yes I als tried limiting the attributes as you suggested but it's not helping.

The odd thing is when I perform the search from the command line on the same search head as show below, the results return immediately.

[root@JHBTNXSPL111 ~]# ldapsearch -x -b "DC=inter,DC=Kransnetwork,DC=Net" -D "CN=SRV-TCC-Splunk LDAP,OU=Service Accounts,OU=Groups,OU=TCC,DC=inter,DC=Kransnetwork,DC=net" -w "I3m5TFfB\$uh2%ze" -h 10.10.227.150 -p 389 -A '(&(objectclass=user)(!(objectClass=computer))(accountExpires=9223372036854775807))'

Also, the Splunk solution uses LDAP integration to authenticate users and it works just fine when users use the AD credentials to log in.

So it really seems to me like the problem is with the "Splunk Supporting Add-on for Active Directory".

Re: How do I improve ldapsearch performance

Lucas_K — Thu, 17 Mar 2016 21:58:27 GMT

There is no doubt the root cause is the addon.

A local os based ldapsearch takes milliseconds. The addon based one takes minutes.

Re: How do I improve ldapsearch performance

gwalford — Thu, 17 Mar 2016 22:39:08 GMT

I opened a ticket, it seems that there is a bug in SA-LDAPsearch that is being addressed.

Re: How do I improve ldapsearch performance

napomokoetle — Fri, 18 Mar 2016 18:30:25 GMT

I agree 100%! There is something NOT optimal with that "Splunk Supporting Add-on for Active Directory" that makes it not perform as expected.

I've seen many other posts from folks experiencing the same problem. The app serves a very useful function, but in its current state it's unusable.

Re: How do I improve ldapsearch performance

napomokoetle — Fri, 18 Mar 2016 18:31:52 GMT

Thank you gwalford! I hope this issue can be finally be resolved. Much appreciated.

Re: How do I improve ldapsearch performance

gwalford — Fri, 18 Mar 2016 21:14:10 GMT

I strongly recommend you open a ticket on this issue as well.

The more customers that open tickets, the higher of a priority this becomes on the DEV side, and the more quickly you will get this issue addressed.

Re: How do I improve ldapsearch performance

napomokoetle — Sat, 19 Mar 2016 12:34:19 GMT

I'll do so.

Re: How do I improve ldapsearch performance

the0duke0 — Fri, 24 Jun 2016 01:56:27 GMT

This issue makes me a very sad panda.

I've had a case open with support since 12/2015 for this issue. They acknowledged that it is an issue they are working on, but still no fix. The problem started with v2 of the add-on. If you go back to the v1 version, which is Java based:-(, the performance is on par with normal LDAP queries. The downside, beyond Java, of downgrading to v1 is that there are lots of features missing. Namely Non-ASCII character support and ability to specify a BaseDN. Going back to v1 broke apps that we use and have created so we are back on slow v2. We try to use scheduled reports as much as possible to hide the slow searching, but that is not ideal, especially for developing searches.